The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

hellaswag

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

# eval: Number of evaluation instances.

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

# train: Number of training instances (e.g., in-context examples).

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

# prompt tokens: Number of tokens in the prompt.

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

# output tokens: Actual number of output tokens.

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

hellaswag_dataset:hellaswag

min=0.765, mean=0.765, max=0.765, sum=0.765 (1)

min=0.217, mean=0.217, max=0.217, sum=0.217 (1)

min=0.726, mean=0.726, max=0.726, sum=0.726 (1)

min=0.614, mean=0.614, max=0.614, sum=0.614 (1)

min=0.284, mean=0.284, max=0.284, sum=0.284 (1)

min=1000, mean=1000, max=1000, sum=1000 (1)

min=62.466, mean=62.466, max=62.466, sum=62.466 (1)

min=0.192, mean=0.192, max=0.192, sum=0.192 (1)

min=0.646, mean=0.646, max=0.646, sum=0.646 (1)

min=0.528, mean=0.528, max=0.528, sum=0.528 (1)

min=0.253, mean=0.253, max=0.253, sum=0.253 (1)

min=0.739, mean=0.739, max=0.739, sum=0.739 (1)

min=0.213, mean=0.213, max=0.213, sum=0.213 (1)

min=0.695, mean=0.695, max=0.695, sum=0.695 (1)

min=0.58, mean=0.58, max=0.58, sum=0.58 (1)

min=0.33, mean=0.33, max=0.33, sum=0.33 (1)

min=0.764, mean=0.764, max=0.764, sum=0.764 (1)

min=0.226, mean=0.226, max=0.226, sum=0.226 (1)

min=0.732, mean=0.732, max=0.732, sum=0.732 (1)

min=0.623, mean=0.623, max=0.623, sum=0.623 (1)

min=0.788, mean=0.788, max=0.788, sum=0.788 (1)

min=0.235, mean=0.235, max=0.235, sum=0.235 (1)

min=0.754, mean=0.754, max=0.754, sum=0.754 (1)

min=0.655, mean=0.655, max=0.655, sum=0.655 (1)

min=0.781, mean=0.781, max=0.781, sum=0.781 (1)

min=0.239, mean=0.239, max=0.239, sum=0.239 (1)

min=0.755, mean=0.755, max=0.755, sum=0.755 (1)

min=0.632, mean=0.632, max=0.632, sum=0.632 (1)

min=0.729, mean=0.729, max=0.729, sum=0.729 (1)

min=0.21, mean=0.21, max=0.21, sum=0.21 (1)

min=0.687, mean=0.687, max=0.687, sum=0.687 (1)

min=0.567, mean=0.567, max=0.567, sum=0.567 (1)

min=0.807, mean=0.807, max=0.807, sum=0.807 (1)

min=0.32, mean=0.32, max=0.32, sum=0.32 (1)

min=0.766, mean=0.766, max=0.766, sum=0.766 (1)

min=0.549, mean=0.549, max=0.549, sum=0.549 (1)

min=87.888, mean=87.888, max=87.888, sum=87.888 (1)

min=1.306, mean=1.306, max=1.306, sum=1.306 (1)

min=0.744, mean=0.744, max=0.744, sum=0.744 (1)

min=0.293, mean=0.293, max=0.293, sum=0.293 (1)

min=0.699, mean=0.699, max=0.699, sum=0.699 (1)

min=0.585, mean=0.585, max=0.585, sum=0.585 (1)

min=0.075, mean=0.075, max=0.075, sum=0.075 (1)

min=88.875, mean=88.875, max=88.875, sum=88.875 (1)

min=0.811, mean=0.811, max=0.811, sum=0.811 (1)

min=0.341, mean=0.341, max=0.341, sum=0.341 (1)

min=0.759, mean=0.759, max=0.759, sum=0.759 (1)

min=0.66, mean=0.66, max=0.66, sum=0.66 (1)

min=0.359, mean=0.359, max=0.359, sum=0.359 (1)

min=88.855, mean=88.855, max=88.855, sum=88.855 (1)

min=0.736, mean=0.736, max=0.736, sum=0.736 (1)

min=0.288, mean=0.288, max=0.288, sum=0.288 (1)

min=0.575, mean=0.575, max=0.575, sum=0.575 (1)

min=0.225, mean=0.225, max=0.225, sum=0.225 (1)

min=0.706, mean=0.706, max=0.706, sum=0.706 (1)

min=0.271, mean=0.271, max=0.271, sum=0.271 (1)

min=0.651, mean=0.651, max=0.651, sum=0.651 (1)

min=0.525, mean=0.525, max=0.525, sum=0.525 (1)

min=0.204, mean=0.204, max=0.204, sum=0.204 (1)

min=0.483, mean=0.483, max=0.483, sum=0.483 (1)

min=0.083, mean=0.083, max=0.083, sum=0.083 (1)

min=0.405, mean=0.405, max=0.405, sum=0.405 (1)

min=0.308, mean=0.308, max=0.308, sum=0.308 (1)

min=0.223, mean=0.223, max=0.223, sum=0.223 (1)

min=0.81, mean=0.81, max=0.81, sum=0.81 (1)

min=0.333, mean=0.333, max=0.333, sum=0.333 (1)

min=0.281, mean=0.281, max=0.281, sum=0.281 (1)

min=0.752, mean=0.752, max=0.752, sum=0.752 (1)

min=0.696, mean=0.696, max=0.696, sum=0.696 (1)

min=0.608, mean=0.608, max=0.608, sum=0.608 (1)

min=0.325, mean=0.325, max=0.325, sum=0.325 (1)

min=0.774, mean=0.774, max=0.774, sum=0.774 (1)

min=0.663, mean=0.663, max=0.663, sum=0.663 (1)

min=0.233, mean=0.233, max=0.233, sum=0.233 (1)

min=0.619, mean=0.619, max=0.619, sum=0.619 (1)

min=0.486, mean=0.486, max=0.486, sum=0.486 (1)

min=0.03, mean=0.03, max=0.03, sum=0.03 (1)

min=0.718, mean=0.718, max=0.718, sum=0.718 (1)

min=0.277, mean=0.277, max=0.277, sum=0.277 (1)

min=0.661, mean=0.661, max=0.661, sum=0.661 (1)

min=0.552, mean=0.552, max=0.552, sum=0.552 (1)

min=0.025, mean=0.025, max=0.025, sum=0.025 (1)

min=88.806, mean=88.806, max=88.806, sum=88.806 (1)

min=0.791, mean=0.791, max=0.791, sum=0.791 (1)

min=0.71, mean=0.71, max=0.71, sum=0.71 (1)

min=0.745, mean=0.745, max=0.745, sum=0.745 (1)

min=0.597, mean=0.597, max=0.597, sum=0.597 (1)

min=0.971, mean=0.971, max=0.971, sum=0.971 (1)

min=0.799, mean=0.799, max=0.799, sum=0.799 (1)

min=0.322, mean=0.322, max=0.322, sum=0.322 (1)

min=0.757, mean=0.757, max=0.757, sum=0.757 (1)

min=0.678, mean=0.678, max=0.678, sum=0.678 (1)

min=0.704, mean=0.704, max=0.704, sum=0.704 (1)

min=0.268, mean=0.268, max=0.268, sum=0.268 (1)

min=0.656, mean=0.656, max=0.656, sum=0.656 (1)

min=0.53, mean=0.53, max=0.53, sum=0.53 (1)

Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.775, mean=0.775, max=0.775, sum=0.775 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.31, mean=0.31, max=0.31, sum=0.31 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.738, mean=0.738, max=0.738, sum=0.738 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.641, mean=0.641, max=0.641, sum=0.641 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.193, mean=0.193, max=0.193, sum=0.193 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=1000, mean=1000, max=1000, sum=1000 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0, mean=0, max=0, sum=0 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=87.888, mean=87.888, max=87.888, sum=87.888 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=1, mean=1, max=1, sum=1 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.682, mean=0.682, max=0.682, sum=0.682 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.25, mean=0.25, max=0.25, sum=0.25 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.632, mean=0.632, max=0.632, sum=0.632 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.522, mean=0.522, max=0.522, sum=0.522 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.084, mean=0.084, max=0.084, sum=0.084 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.555, mean=0.555, max=0.555, sum=0.555 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.144, mean=0.144, max=0.144, sum=0.144 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.489, mean=0.489, max=0.489, sum=0.489 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.401, mean=0.401, max=0.401, sum=0.401 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.113, mean=0.113, max=0.113, sum=0.113 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.435, mean=0.435, max=0.435, sum=0.435 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.057, mean=0.057, max=0.057, sum=0.057 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.37, mean=0.37, max=0.37, sum=0.37 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.294, mean=0.294, max=0.294, sum=0.294 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.138, mean=0.138, max=0.138, sum=0.138 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.822, mean=0.822, max=0.822, sum=0.822 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.278, mean=0.278, max=0.278, sum=0.278 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.798, mean=0.798, max=0.798, sum=0.798 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.729, mean=0.729, max=0.729, sum=0.729 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.815, mean=0.815, max=0.815, sum=0.815 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.286, mean=0.286, max=0.286, sum=0.286 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.776, mean=0.776, max=0.776, sum=0.776 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.703, mean=0.703, max=0.703, sum=0.703 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.171, mean=0.171, max=0.171, sum=0.171 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.676, mean=0.676, max=0.676, sum=0.676 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.153, mean=0.153, max=0.153, sum=0.153 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.625, mean=0.625, max=0.625, sum=0.625 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.534, mean=0.534, max=0.534, sum=0.534 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.125, mean=0.125, max=0.125, sum=0.125 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.561, mean=0.561, max=0.561, sum=0.561 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.083, mean=0.083, max=0.083, sum=0.083 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.468, mean=0.468, max=0.468, sum=0.468 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.405, mean=0.405, max=0.405, sum=0.405 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.429, mean=0.429, max=0.429, sum=0.429 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.103, mean=0.103, max=0.103, sum=0.103 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.32, mean=0.32, max=0.32, sum=0.32 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.27, mean=0.27, max=0.27, sum=0.27 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.079, mean=0.079, max=0.079, sum=0.079 (1)
⚠ Brown et al. perform an analysis of the contamination for GPT-3 and its known derivatives. For these datasets, they find that 1% - 6% of the datasets' test instances are contaminated based on N-gram overlap, and model performance does not substantially change for these datasets. See Table C.1 on page 45 of https://arxiv.org/pdf/2005.14165.pdf.

min=0.25, mean=0.25, max=0.25, sum=0.25 (1)