The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

natural_qa_openbook_longans

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

# eval: Number of evaluation instances.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

# train: Number of training instances (e.g., in-context examples).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

# prompt tokens: Number of tokens in the prompt.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

# output tokens: Actual number of output tokens.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

natural_qa_openbook_longans_mode:openbook_longans

min=0.593, mean=0.595, max=0.598, sum=1.786 (3)

min=0.058, mean=0.065, max=0.069, sum=0.195 (3)

min=0.487, mean=0.503, max=0.515, sum=1.509 (3)

min=0.54, mean=0.54, max=0.54, sum=1.62 (3)

min=0.667, mean=0.667, max=0.667, sum=2 (3)

min=0.38, mean=0.46, max=0.5, sum=1.38 (3)

min=0.451, mean=0.478, max=0.506, sum=1.433 (3)

min=0.011, mean=0.041, max=0.063, sum=0.122 (3)

min=0.001, mean=0.001, max=0.002, sum=0.004 (3)

min=0.931, mean=1.06, max=1.147, sum=3.179 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=4.568, mean=4.666, max=4.734, sum=13.999 (3)

min=0.038, mean=0.038, max=0.038, sum=0.114 (3)

min=1136.933, mean=1418.457, max=1595.508, sum=4255.37 (3)

min=5.418, mean=5.682, max=5.988, sum=17.046 (3)

min=0.5, mean=0.532, max=0.571, sum=1.597 (3)

min=0.075, mean=0.086, max=0.093, sum=0.258 (3)

min=0.361, mean=0.41, max=0.455, sum=1.23 (3)

min=0.44, mean=0.47, max=0.508, sum=1.409 (3)

min=0.667, mean=0.667, max=0.667, sum=1.333 (2)

min=0.327, mean=0.394, max=0.457, sum=1.182 (3)

min=0.106, mean=0.109, max=0.113, sum=0.328 (3)

min=0, mean=0.001, max=0.001, sum=0.002 (3)

min=0.66, mean=0.733, max=0.784, sum=2.198 (3)

min=5.487, mean=5.946, max=6.338, sum=17.838 (3)

min=0.561, mean=0.578, max=0.59, sum=1.734 (3)

min=0.073, mean=0.081, max=0.097, sum=0.243 (3)

min=0.449, mean=0.478, max=0.494, sum=1.433 (3)

min=0.501, mean=0.521, max=0.534, sum=1.563 (3)

min=0.667, mean=0.667, max=0.667, sum=0.667 (1)

min=0.346, mean=0.346, max=0.346, sum=1.038 (3)

min=0.45, mean=0.488, max=0.521, sum=1.463 (3)

min=0.111, mean=0.113, max=0.118, sum=0.34 (3)

min=0.774, mean=0.873, max=0.927, sum=2.618 (3)

min=6.302, mean=6.538, max=6.976, sum=19.615 (3)

min=0.622, mean=0.625, max=0.628, sum=1.874 (3)

min=0.051, mean=0.065, max=0.075, sum=0.196 (3)

min=0.556, mean=0.56, max=0.568, sum=1.681 (3)

min=0.562, mean=0.571, max=0.578, sum=1.714 (3)

min=0.1, mean=0.167, max=0.3, sum=0.5 (3)

min=0.478, mean=0.488, max=0.498, sum=1.465 (3)

min=0.369, mean=0.381, max=0.394, sum=1.143 (3)

min=0.001, mean=0.001, max=0.001, sum=0.003 (3)

min=5.132, mean=5.27, max=5.521, sum=15.809 (3)

min=0.65, mean=0.669, max=0.681, sum=2.007 (3)

min=0.071, mean=0.073, max=0.076, sum=0.22 (3)

min=0.576, mean=0.599, max=0.616, sum=1.796 (3)

min=0.601, mean=0.62, max=0.633, sum=1.86 (3)

min=0.3, mean=0.413, max=0.5, sum=1.238 (3)

min=0.514, mean=0.541, max=0.561, sum=1.624 (3)

min=0.06, mean=0.107, max=0.132, sum=0.321 (3)

min=4.928, mean=4.93, max=4.932, sum=14.791 (3)

min=0.012, mean=0.012, max=0.012, sum=0.036 (3)

min=1254.565, mean=1571.171, max=1771.274, sum=4713.512 (3)

min=4.785, mean=5.113, max=5.399, sum=15.338 (3)

min=0.627, mean=0.639, max=0.649, sum=1.917 (3)

min=0.055, mean=0.063, max=0.072, sum=0.189 (3)

min=0.555, mean=0.563, max=0.567, sum=1.69 (3)

min=0.569, mean=0.584, max=0.592, sum=1.751 (3)

min=0.431, mean=0.465, max=0.498, sum=1.395 (3)

min=0.01, mean=0.031, max=0.053, sum=0.093 (3)

min=0, mean=0.0, max=0.001, sum=0.001 (3)

min=5.26, mean=5.588, max=6.066, sum=16.764 (3)

min=0.576, mean=0.589, max=0.605, sum=1.766 (3)

min=0.081, mean=0.084, max=0.089, sum=0.253 (3)

min=0.485, mean=0.503, max=0.529, sum=1.51 (3)

min=0.53, mean=0.539, max=0.557, sum=1.616 (3)

min=0.3, mean=0.433, max=0.5, sum=1.3 (3)

min=0.441, mean=0.504, max=0.574, sum=1.513 (3)

min=0.052, mean=0.158, max=0.225, sum=0.474 (3)

min=0.001, mean=0.002, max=0.003, sum=0.006 (3)

min=5.785, mean=6.297, max=6.845, sum=18.892 (3)

min=0.563, mean=0.568, max=0.577, sum=1.705 (3)

min=0.068, mean=0.07, max=0.074, sum=0.21 (3)

min=0.484, mean=0.491, max=0.498, sum=1.474 (3)

min=0.505, mean=0.511, max=0.515, sum=1.534 (3)

min=0.3, mean=0.432, max=0.5, sum=1.296 (3)

min=0.429, mean=0.457, max=0.498, sum=1.37 (3)

min=0.272, mean=0.32, max=0.416, sum=0.961 (3)

min=0, mean=0.001, max=0.003, sum=0.004 (3)

min=4.691, mean=4.711, max=4.726, sum=14.134 (3)

min=0.038, mean=0.039, max=0.04, sum=0.116 (3)

min=1224.733, mean=1384.565, max=1488.14, sum=4153.695 (3)

min=7.685, mean=10.15, max=11.898, sum=30.449 (3)

min=0.606, mean=0.609, max=0.611, sum=1.827 (3)

min=0.08, mean=0.09, max=0.095, sum=0.269 (3)

min=0.515, mean=0.524, max=0.537, sum=1.572 (3)

min=0.548, mean=0.551, max=0.554, sum=1.654 (3)

min=0.4, mean=0.411, max=0.433, sum=1.233 (3)

min=0.421, mean=0.441, max=0.477, sum=1.324 (3)

min=0.022, mean=0.045, max=0.082, sum=0.135 (3)

min=0, mean=0.001, max=0.002, sum=0.003 (3)

min=8.216, mean=10.3, max=11.913, sum=30.9 (3)

min=0.644, mean=0.649, max=0.656, sum=1.946 (3)

min=0.07, mean=0.074, max=0.077, sum=0.222 (3)

min=0.576, mean=0.586, max=0.593, sum=1.758 (3)

min=0.583, mean=0.597, max=0.61, sum=1.79 (3)

min=0.4, mean=0.444, max=0.5, sum=1.333 (3)

min=0.401, mean=0.44, max=0.506, sum=1.319 (3)

min=0.205, mean=0.22, max=0.25, sum=0.66 (3)

min=0, mean=0.001, max=0.002, sum=0.004 (3)

min=6.064, mean=6.362, max=6.864, sum=19.086 (3)

min=0.682, mean=0.686, max=0.693, sum=2.059 (3)

min=0.048, mean=0.067, max=0.088, sum=0.2 (3)

min=0.608, mean=0.632, max=0.646, sum=1.897 (3)

min=0.639, mean=0.642, max=0.646, sum=1.927 (3)

min=0.367, mean=0.429, max=0.5, sum=1.287 (3)

min=0.459, mean=0.48, max=0.498, sum=1.441 (3)

min=0.024, mean=0.043, max=0.079, sum=0.129 (3)

min=1.036, mean=1.102, max=1.15, sum=3.305 (3)

min=4.964, mean=4.964, max=4.965, sum=14.893 (3)

min=0.007, mean=0.007, max=0.007, sum=0.021 (3)

min=1381.066, mean=1592.701, max=1704.681, sum=4778.103 (3)

min=5.429, mean=5.659, max=6.028, sum=16.976 (3)

min=0.61, mean=0.621, max=0.628, sum=1.864 (3)

min=0.338, mean=0.347, max=0.36, sum=1.041 (3)

min=0.547, mean=0.558, max=0.569, sum=1.675 (3)

min=0.56, mean=0.575, max=0.585, sum=1.724 (3)

min=0.352, mean=0.426, max=0.5, sum=0.852 (2)

min=0.454, mean=0.499, max=0.546, sum=1.498 (3)

min=0.11, mean=0.135, max=0.177, sum=0.404 (3)

min=2.213, mean=2.547, max=2.912, sum=7.64 (3)

min=4.73, mean=4.743, max=4.751, sum=14.229 (3)

min=0.035, mean=0.035, max=0.035, sum=0.105 (3)

min=1193.69, mean=1313.422, max=1423.457, sum=3940.267 (3)

min=31.304, mean=38.803, max=46.481, sum=116.409 (3)

min=0.171, mean=0.19, max=0.203, sum=0.569 (3)

min=0.101, mean=0.122, max=0.135, sum=0.367 (3)

min=0.119, mean=0.136, max=0.151, sum=0.407 (3)

min=0.209, mean=0.329, max=0.473, sum=0.987 (3)

min=0.289, mean=0.388, max=0.456, sum=1.164 (3)

min=0.394, mean=0.462, max=0.563, sum=1.386 (3)

min=0.044, mean=0.091, max=0.176, sum=0.273 (3)

min=2.864, mean=2.895, max=2.953, sum=8.685 (3)

min=3.164, mean=3.396, max=3.709, sum=10.189 (3)

min=0.052, mean=0.057, max=0.066, sum=0.172 (3)

min=850.863, mean=903.877, max=958.904, sum=2711.631 (3)

min=0.576, mean=0.595, max=0.607, sum=1.785 (3)

min=0.075, mean=0.085, max=0.099, sum=0.254 (3)

min=0.411, mean=0.471, max=0.518, sum=1.414 (3)

min=0.521, mean=0.535, max=0.546, sum=1.604 (3)

min=0.332, mean=0.388, max=0.488, sum=1.163 (3)

min=0.373, mean=0.409, max=0.446, sum=1.226 (3)

min=0.026, mean=0.051, max=0.066, sum=0.153 (3)

min=0.95, mean=1.085, max=1.249, sum=3.256 (3)

min=4.538, mean=4.633, max=4.715, sum=13.899 (3)

min=0.039, mean=0.039, max=0.039, sum=0.117 (3)

min=1261.72, mean=1481.344, max=1608.455, sum=4444.032 (3)

min=7.154, mean=8.834, max=11.932, sum=26.502 (3)

min=0.553, mean=0.573, max=0.584, sum=1.72 (3)

min=0.117, mean=0.143, max=0.158, sum=0.43 (3)

min=0.287, mean=0.347, max=0.38, sum=1.041 (3)

min=0.489, mean=0.507, max=0.516, sum=1.52 (3)

min=0.37, mean=0.39, max=0.4, sum=1.17 (3)

min=0.447, mean=0.457, max=0.467, sum=1.371 (3)

min=0.125, mean=0.174, max=0.251, sum=0.523 (3)

min=0.001, mean=0.002, max=0.003, sum=0.005 (3)

min=0.681, mean=0.774, max=0.827, sum=2.321 (3)

min=8.71, mean=10.443, max=11.438, sum=31.329 (3)

min=0.482, mean=0.504, max=0.516, sum=1.512 (3)

min=0.129, mean=0.142, max=0.154, sum=0.425 (3)

min=0.226, mean=0.266, max=0.292, sum=0.799 (3)

min=0.41, mean=0.432, max=0.444, sum=1.297 (3)

min=0.292, mean=0.354, max=0.417, sum=0.708 (2)

min=0.289, mean=0.325, max=0.385, sum=0.974 (3)

min=0.202, mean=0.234, max=0.285, sum=0.703 (3)

min=0.002, mean=0.002, max=0.003, sum=0.007 (3)

min=0.476, mean=0.535, max=0.583, sum=1.606 (3)

min=7.288, mean=9.101, max=11.307, sum=27.304 (3)

min=0.291, mean=0.309, max=0.334, sum=0.928 (3)

min=0.18, mean=0.198, max=0.221, sum=0.594 (3)

min=0.066, mean=0.074, max=0.08, sum=0.222 (3)

min=0.198, mean=0.219, max=0.246, sum=0.657 (3)

min=0.48, mean=0.485, max=0.494, sum=1.455 (3)

min=0.382, mean=0.435, max=0.467, sum=1.306 (3)

min=0.234, mean=0.265, max=0.3, sum=0.796 (3)

min=0.002, mean=0.003, max=0.003, sum=0.008 (3)

min=0.552, mean=0.605, max=0.643, sum=1.815 (3)

min=20.452, mean=22.835, max=25.41, sum=68.505 (3)

min=0.619, mean=0.628, max=0.634, sum=1.885 (3)

min=0.064, mean=0.073, max=0.08, sum=0.219 (3)

min=0.49, mean=0.533, max=0.555, sum=1.598 (3)