The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

natural_qa_closedbook

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

# eval: Number of evaluation instances.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

# train: Number of training instances (e.g., in-context examples).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

# prompt tokens: Number of tokens in the prompt.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

# output tokens: Actual number of output tokens.

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

natural_qa_closedbook_mode:closedbook

min=0.288, mean=0.293, max=0.302, sum=0.879 (3)

min=0.029, mean=0.035, max=0.042, sum=0.106 (3)

min=0.177, mean=0.179, max=0.183, sum=0.537 (3)

min=0.227, mean=0.235, max=0.239, sum=0.704 (3)

min=0.255, mean=0.333, max=0.386, sum=1.0 (3)

min=0.125, mean=0.175, max=0.2, sum=0.525 (3)

min=0.466, mean=0.493, max=0.536, sum=1.478 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=94.377, mean=99.377, max=102.377, sum=298.131 (3)

min=5.012, mean=5.602, max=6.608, sum=16.806 (3)

min=0.182, mean=0.19, max=0.196, sum=0.571 (3)

min=0.013, mean=0.015, max=0.017, sum=0.046 (3)

min=0.092, mean=0.098, max=0.106, sum=0.293 (3)

min=0.14, mean=0.146, max=0.151, sum=0.439 (3)

min=0.214, mean=0.405, max=0.5, sum=1.214 (3)

min=0.302, mean=0.362, max=0.45, sum=1.085 (3)

min=0.088, mean=0.216, max=0.371, sum=0.647 (3)

min=0.001, mean=0.002, max=0.002, sum=0.005 (3)

min=0.355, mean=0.372, max=0.396, sum=1.117 (3)

min=6.868, mean=7.876, max=9.311, sum=23.628 (3)

min=0.229, mean=0.233, max=0.239, sum=0.7 (3)

min=0.027, mean=0.029, max=0.03, sum=0.087 (3)

min=0.164, mean=0.17, max=0.175, sum=0.511 (3)

min=0.185, mean=0.187, max=0.189, sum=0.562 (3)

min=0.473, mean=0.521, max=0.556, sum=1.564 (3)

min=0.437, mean=0.466, max=0.494, sum=1.399 (3)

min=4.791, mean=5.971, max=7.18, sum=17.913 (3)

min=0.332, mean=0.337, max=0.341, sum=1.012 (3)

min=0.031, mean=0.036, max=0.043, sum=0.107 (3)

min=0.23, mean=0.235, max=0.241, sum=0.705 (3)

min=0.269, mean=0.27, max=0.27, sum=0.81 (3)

min=0.667, mean=0.667, max=0.667, sum=1.333 (2)

min=0.288, mean=0.392, max=0.491, sum=1.177 (3)

min=0.026, mean=0.174, max=0.318, sum=0.522 (3)

min=0.001, mean=0.001, max=0.002, sum=0.004 (3)

min=4.471, mean=5.282, max=6.145, sum=15.846 (3)

min=0.384, mean=0.385, max=0.387, sum=1.156 (3)

min=0.018, mean=0.018, max=0.019, sum=0.054 (3)

min=0.31, mean=0.315, max=0.318, sum=0.945 (3)

min=0.326, mean=0.327, max=0.328, sum=0.982 (3)

min=0.352, mean=0.376, max=0.405, sum=1.127 (3)

min=0.033, mean=0.095, max=0.136, sum=0.285 (3)

min=0, mean=0.001, max=0.001, sum=0.002 (3)

min=5.127, mean=5.365, max=5.79, sum=16.095 (3)

min=0.35, mean=0.356, max=0.362, sum=1.068 (3)

min=0.016, mean=0.018, max=0.019, sum=0.054 (3)

min=0.277, mean=0.285, max=0.29, sum=0.854 (3)

min=0.276, mean=0.283, max=0.288, sum=0.85 (3)

min=0.667, mean=0.667, max=0.667, sum=2 (3)

min=0.431, mean=0.507, max=0.569, sum=1.52 (3)

min=0.1, mean=0.176, max=0.273, sum=0.527 (3)

min=0, mean=0.0, max=0.001, sum=0.001 (3)

min=5.466, mean=6.315, max=6.864, sum=18.944 (3)

min=0.269, mean=0.274, max=0.28, sum=0.821 (3)

min=0.008, mean=0.014, max=0.021, sum=0.042 (3)

min=0.177, mean=0.187, max=0.195, sum=0.562 (3)

min=0.215, mean=0.217, max=0.221, sum=0.652 (3)

min=0.502, mean=0.531, max=0.563, sum=1.594 (3)

min=0, mean=0.079, max=0.192, sum=0.238 (3)

min=0.001, mean=0.001, max=0.001, sum=0.003 (3)

min=5.924, mean=6.729, max=7.956, sum=20.187 (3)

min=0.197, mean=0.202, max=0.206, sum=0.606 (3)

min=0.039, mean=0.045, max=0.054, sum=0.136 (3)

min=0.157, mean=0.163, max=0.168, sum=0.489 (3)

min=0.156, mean=0.16, max=0.164, sum=0.481 (3)

min=0.25, mean=0.417, max=0.5, sum=1.25 (3)

min=0.339, mean=0.433, max=0.5, sum=1.298 (3)

min=0.081, mean=0.162, max=0.239, sum=0.486 (3)

min=0.002, mean=0.002, max=0.002, sum=0.006 (3)

min=109.087, mean=111.754, max=116.087, sum=335.261 (3)

min=4.314, mean=5.287, max=5.908, sum=15.861 (3)

min=0.252, mean=0.254, max=0.257, sum=0.762 (3)

min=0.018, mean=0.022, max=0.024, sum=0.065 (3)

min=0.205, mean=0.212, max=0.218, sum=0.635 (3)

min=0.205, mean=0.214, max=0.22, sum=0.642 (3)

min=0.667, mean=0.667, max=0.667, sum=0.667 (1)

min=0.379, mean=0.46, max=0.5, sum=1.379 (3)

min=0.414, mean=0.435, max=0.447, sum=1.304 (3)

min=0.15, mean=0.223, max=0.269, sum=0.669 (3)

min=5.508, mean=6.119, max=6.869, sum=18.356 (3)

min=0.281, mean=0.293, max=0.299, sum=0.878 (3)

min=0.035, mean=0.041, max=0.045, sum=0.123 (3)

min=0.243, mean=0.252, max=0.261, sum=0.757 (3)

min=0.23, mean=0.241, max=0.25, sum=0.723 (3)

min=0.338, mean=0.446, max=0.5, sum=1.338 (3)

min=0.467, mean=0.48, max=0.498, sum=1.441 (3)

min=0.02, mean=0.125, max=0.265, sum=0.374 (3)

min=0.001, mean=0.002, max=0.003, sum=0.006 (3)

min=4.262, mean=4.508, max=4.666, sum=13.525 (3)

min=0.279, mean=0.288, max=0.295, sum=0.863 (3)

min=0.108, mean=0.121, max=0.128, sum=0.362 (3)

min=0.239, mean=0.245, max=0.248, sum=0.734 (3)

min=0.228, mean=0.239, max=0.244, sum=0.716 (3)

min=0.358, mean=0.386, max=0.439, sum=1.158 (3)

min=0, mean=0.148, max=0.237, sum=0.443 (3)

min=0.751, mean=0.777, max=0.821, sum=2.331 (3)

min=110.254, mean=112.254, max=116.254, sum=336.762 (3)

min=5.032, mean=5.47, max=6.183, sum=16.409 (3)

min=0.208, mean=0.216, max=0.221, sum=0.648 (3)

min=0.11, mean=0.116, max=0.118, sum=0.347 (3)

min=0.18, mean=0.185, max=0.19, sum=0.556 (3)

min=0.183, mean=0.187, max=0.189, sum=0.56 (3)

min=0.303, mean=0.418, max=0.519, sum=1.254 (3)

min=0.056, mean=0.09, max=0.143, sum=0.27 (3)

min=0.931, mean=1.115, max=1.261, sum=3.346 (3)

min=92.12, mean=96.12, max=102.12, sum=288.36 (3)

min=34.82, mean=48.109, max=57.074, sum=144.327 (3)

min=0.038, mean=0.039, max=0.04, sum=0.117 (3)

min=0.03, mean=0.031, max=0.032, sum=0.092 (3)

min=0.027, mean=0.028, max=0.03, sum=0.084 (3)

min=0.385, mean=0.462, max=0.5, sum=1.385 (3)

min=0.552, mean=0.613, max=0.657, sum=1.84 (3)

min=0.028, mean=0.177, max=0.252, sum=0.53 (3)

min=1.309, mean=1.457, max=1.621, sum=4.371 (3)

min=109.556, mean=113.556, max=118.556, sum=340.668 (3)

min=0.302, mean=0.312, max=0.32, sum=0.937 (3)

min=0.061, mean=0.068, max=0.073, sum=0.203 (3)

min=0.233, mean=0.238, max=0.241, sum=0.713 (3)

min=0.248, mean=0.255, max=0.259, sum=0.764 (3)

min=0.382, mean=0.43, max=0.498, sum=1.291 (3)

min=0.024, mean=0.094, max=0.18, sum=0.281 (3)

min=0.546, mean=0.565, max=0.586, sum=1.694 (3)

min=109.191, mean=111.191, max=115.191, sum=333.573 (3)

min=5.31, mean=5.844, max=6.407, sum=17.531 (3)

min=0.227, mean=0.232, max=0.235, sum=0.695 (3)

min=0.02, mean=0.025, max=0.032, sum=0.074 (3)

min=0.16, mean=0.172, max=0.18, sum=0.515 (3)

min=0.176, mean=0.178, max=0.181, sum=0.535 (3)

min=0.244, mean=0.34, max=0.429, sum=1.021 (3)

min=0.208, mean=0.233, max=0.269, sum=0.7 (3)

min=0.332, mean=0.337, max=0.343, sum=1.012 (3)

min=5.441, mean=5.625, max=5.917, sum=16.875 (3)

min=0.173, mean=0.177, max=0.179, sum=0.53 (3)

min=0.018, mean=0.026, max=0.036, sum=0.077 (3)

min=0.097, mean=0.102, max=0.104, sum=0.305 (3)

min=0.124, mean=0.126, max=0.127, sum=0.377 (3)

min=0.419, mean=0.441, max=0.476, sum=1.323 (3)

min=0.214, mean=0.251, max=0.3, sum=0.753 (3)

min=0.254, mean=0.259, max=0.265, sum=0.778 (3)

min=4.823, mean=5.267, max=5.728, sum=15.801 (3)

min=0.072, mean=0.078, max=0.082, sum=0.234 (3)

min=0.02, mean=0.023, max=0.027, sum=0.07 (3)

min=0.024, mean=0.025, max=0.027, sum=0.075 (3)

min=0.052, mean=0.055, max=0.062, sum=0.166 (3)

min=0.238, mean=0.415, max=0.539, sum=1.244 (3)

min=0.167, mean=0.234, max=0.286, sum=0.702 (3)

min=0.246, mean=0.251, max=0.259, sum=0.753 (3)

min=4.325, mean=5.149, max=6.46, sum=15.446 (3)

min=0.359, mean=0.361, max=0.365, sum=1.083 (3)

min=0.044, mean=0.054, max=0.064, sum=0.163 (3)

min=0.295, mean=0.299, max=0.303, sum=0.898 (3)

min=0.333, mean=0.444, max=0.5, sum=1.333 (3)

min=0.407, mean=0.48, max=0.556, sum=1.441 (3)

min=0.147, mean=0.247, max=0.385, sum=0.741 (3)

min=4.528, mean=4.808, max=5.211, sum=14.424 (3)

min=0.193, mean=0.199, max=0.203, sum=0.598 (3)

min=0.006, mean=0.015, max=0.02, sum=0.044 (3)

min=0.099, mean=0.105, max=0.11, sum=0.314 (3)

min=0.142, mean=0.149, max=0.157, sum=0.447 (3)

min=0.357, mean=0.45, max=0.5, sum=1.349 (3)

min=0.382, mean=0.451, max=0.504, sum=1.353 (3)

min=0.173, mean=0.314, max=0.386, sum=0.942 (3)

min=6.631, mean=6.745, max=6.831, sum=20.236 (3)

min=0.227, mean=0.229, max=0.233, sum=0.687 (3)

min=0.04, mean=0.042, max=0.046, sum=0.127 (3)

min=0.156, mean=0.163, max=0.171, sum=0.489 (3)

min=0.165, mean=0.167, max=0.167, sum=0.5 (3)

min=0.421, mean=0.465, max=0.506, sum=1.394 (3)

min=0.1, mean=0.183, max=0.3, sum=0.55 (3)

min=4.428, mean=4.687, max=4.995, sum=14.06 (3)

min=0.369, mean=0.372, max=0.374, sum=1.116 (3)

min=0.075, mean=0.084, max=0.091, sum=0.251 (3)

min=0.286, mean=0.289, max=0.294, sum=0.867 (3)

min=0.29, mean=0.296, max=0.301, sum=0.888 (3)

min=0.487, mean=0.552, max=0.634, sum=1.655 (3)

min=0.063, mean=0.129, max=0.206, sum=0.387 (3)

min=4.29, mean=4.325, max=4.367, sum=12.974 (3)

min=0.146, mean=0.156, max=0.164, sum=0.468 (3)

min=0.069, mean=0.075, max=0.079, sum=0.224 (3)

min=0.09, mean=0.099, max=0.109, sum=0.298 (3)

min=0.112, mean=0.122, max=0.128, sum=0.365 (3)

min=0.373, mean=0.49, max=0.553, sum=1.47 (3)