The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

narrative_qa

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

F1: Average F1 score in terms of word overlap between the model output and correct reference.

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

# eval: Number of evaluation instances.

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

# train: Number of training instances (e.g., in-context examples).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

# prompt tokens: Number of tokens in the prompt.

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

# output tokens: Actual number of output tokens.

The NarrativeQA benchmark for reading comprehension over narratives [(Kočiský et al., 2017)](https://aclanthology.org/Q18-1023/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

narrative_qa_

min=0.689, mean=0.695, max=0.698, sum=2.085 (3)

min=0.028, mean=0.034, max=0.042, sum=0.101 (3)

min=0.448, mean=0.523, max=0.573, sum=1.57 (3)

min=0.566, mean=0.581, max=0.592, sum=1.743 (3)

min=0.375, mean=0.438, max=0.5, sum=0.875 (2)

min=0.667, mean=0.667, max=0.667, sum=1.333 (2)

min=0.196, mean=0.214, max=0.225, sum=0.641 (3)

min=0.014, mean=0.014, max=0.014, sum=0.042 (3)

min=1.085, mean=1.126, max=1.167, sum=3.379 (3)

min=2.166, mean=2.639, max=3.225, sum=7.918 (3)

min=1598.614, mean=1692.218, max=1777.299, sum=5076.654 (3)

min=4.434, mean=4.514, max=4.617, sum=13.541 (3)

min=0.612, mean=0.623, max=0.634, sum=1.87 (3)

min=0.042, mean=0.046, max=0.048, sum=0.137 (3)

min=0.341, mean=0.4, max=0.438, sum=1.201 (3)

min=0.496, mean=0.513, max=0.524, sum=1.538 (3)

min=0.667, mean=0.667, max=0.667, sum=2 (3)

min=0.17, mean=0.203, max=0.223, sum=0.609 (3)

min=0.011, mean=0.013, max=0.014, sum=0.039 (3)

min=0.768, mean=0.797, max=0.829, sum=2.391 (3)

min=4.797, mean=5.09, max=5.518, sum=15.27 (3)

min=0.664, mean=0.672, max=0.68, sum=2.016 (3)

min=0.039, mean=0.047, max=0.062, sum=0.141 (3)

min=0.409, mean=0.477, max=0.522, sum=1.432 (3)

min=0.526, mean=0.547, max=0.563, sum=1.641 (3)

min=0.15, mean=0.164, max=0.18, sum=0.491 (3)

min=0.008, mean=0.014, max=0.017, sum=0.042 (3)

min=0.892, mean=0.923, max=0.955, sum=2.769 (3)

min=4.324, mean=4.528, max=4.701, sum=13.583 (3)

min=0.712, mean=0.725, max=0.736, sum=2.176 (3)

min=0.034, mean=0.041, max=0.05, sum=0.122 (3)

min=0.484, mean=0.565, max=0.616, sum=1.694 (3)

min=0.622, mean=0.647, max=0.665, sum=1.941 (3)

min=0.667, mean=0.667, max=0.667, sum=0.667 (1)

min=0.156, mean=0.179, max=0.205, sum=0.536 (3)

min=4.194, mean=4.6, max=5.011, sum=13.8 (3)

min=0.387, mean=0.404, max=0.432, sum=1.212 (3)

min=0.134, mean=0.193, max=0.229, sum=0.58 (3)

min=0.321, mean=0.331, max=0.346, sum=0.993 (3)

min=0.318, mean=0.335, max=0.362, sum=1.004 (3)

min=0.333, mean=0.377, max=0.429, sum=1.131 (3)

min=0.161, mean=0.196, max=0.216, sum=0.588 (3)

min=0.008, mean=0.01, max=0.011, sum=0.031 (3)

min=2548.434, mean=2833.767, max=3041.434, sum=8501.301 (3)

min=8.966, mean=10.559, max=11.732, sum=31.676 (3)

min=0.386, mean=0.407, max=0.442, sum=1.221 (3)

min=0.1, mean=0.199, max=0.279, sum=0.596 (3)

min=0.298, mean=0.308, max=0.313, sum=0.924 (3)

min=0.322, mean=0.337, max=0.365, sum=1.011 (3)

min=0, mean=0.238, max=0.405, sum=0.714 (3)

min=0.165, mean=0.181, max=0.189, sum=0.542 (3)

min=0.008, mean=0.01, max=0.014, sum=0.031 (3)

min=2.149, mean=2.628, max=3.22, sum=7.885 (3)

min=1598.411, mean=1692.838, max=1775.994, sum=5078.515 (3)

min=8.304, mean=8.974, max=9.338, sum=26.921 (3)

min=0.344, mean=0.378, max=0.426, sum=1.135 (3)

min=0.065, mean=0.131, max=0.217, sum=0.393 (3)

min=0.267, mean=0.272, max=0.278, sum=0.817 (3)

min=0.259, mean=0.293, max=0.339, sum=0.88 (3)

min=0.375, mean=0.403, max=0.417, sum=1.208 (3)

min=0.123, mean=0.144, max=0.156, sum=0.431 (3)

min=0.008, mean=0.013, max=0.017, sum=0.039 (3)

min=9.687, mean=10.07, max=10.651, sum=30.211 (3)

min=0.577, mean=0.605, max=0.633, sum=1.815 (3)

min=0.04, mean=0.048, max=0.063, sum=0.145 (3)

min=0.444, mean=0.476, max=0.505, sum=1.429 (3)

min=0.462, mean=0.498, max=0.532, sum=1.495 (3)

min=0.396, mean=0.438, max=0.5, sum=1.313 (3)

min=0.333, mean=0.556, max=0.667, sum=1.667 (3)

min=0.152, mean=0.172, max=0.197, sum=0.516 (3)

min=0.02, mean=0.022, max=0.025, sum=0.065 (3)

min=1.039, mean=1.621, max=2.037, sum=4.862 (3)

min=1606.952, mean=1647.783, max=1694.642, sum=4943.349 (3)

min=5.521, mean=6.798, max=8.192, sum=20.394 (3)

min=0.637, mean=0.665, max=0.684, sum=1.994 (3)

min=0.043, mean=0.046, max=0.047, sum=0.138 (3)

min=0.481, mean=0.513, max=0.539, sum=1.54 (3)

min=0.503, mean=0.532, max=0.565, sum=1.597 (3)

min=0.4, mean=0.416, max=0.44, sum=1.248 (3)

min=0.186, mean=0.199, max=0.207, sum=0.598 (3)

min=0.014, mean=0.017, max=0.02, sum=0.051 (3)

min=6.321, mean=7.042, max=8.175, sum=21.127 (3)

min=0.687, mean=0.711, max=0.742, sum=2.133 (3)

min=0.036, mean=0.049, max=0.061, sum=0.147 (3)

min=0.557, mean=0.59, max=0.617, sum=1.771 (3)

min=0.562, mean=0.603, max=0.637, sum=1.808 (3)

min=0.396, mean=0.465, max=0.5, sum=1.396 (3)

min=0.216, mean=0.238, max=0.256, sum=0.714 (3)

min=0.011, mean=0.016, max=0.02, sum=0.048 (3)

min=5.749, mean=6.84, max=8.158, sum=20.521 (3)

min=0.692, mean=0.728, max=0.748, sum=2.185 (3)

min=0.052, mean=0.09, max=0.14, sum=0.27 (3)

min=0.622, mean=0.663, max=0.693, sum=1.99 (3)

min=0.614, mean=0.646, max=0.667, sum=1.939 (3)

min=0.333, mean=0.39, max=0.419, sum=1.169 (3)

min=0.19, mean=0.208, max=0.218, sum=0.624 (3)

min=1.628, mean=1.722, max=1.839, sum=5.167 (3)

min=3504.577, mean=3803.911, max=3972.577, sum=11411.732 (3)

min=4.572, mean=6.952, max=8.434, sum=20.856 (3)

min=0.631, mean=0.662, max=0.695, sum=1.986 (3)

min=0.231, mean=0.237, max=0.242, sum=0.712 (3)

min=0.468, mean=0.53, max=0.574, sum=1.591 (3)

min=0.535, mean=0.577, max=0.613, sum=1.73 (3)

min=0.333, mean=0.355, max=0.389, sum=1.065 (3)

min=0.118, mean=0.165, max=0.241, sum=0.494 (3)

min=0.011, mean=0.012, max=0.014, sum=0.037 (3)

min=2.081, mean=2.598, max=3.427, sum=7.794 (3)

min=1.042, mean=1.621, max=2.048, sum=4.862 (3)

min=1604.899, mean=1649.598, max=1699.146, sum=4948.794 (3)

min=18.468, mean=33.276, max=50.499, sum=99.828 (3)

min=0.139, mean=0.151, max=0.158, sum=0.454 (3)

min=0.087, mean=0.099, max=0.105, sum=0.296 (3)

min=0.074, mean=0.086, max=0.093, sum=0.258 (3)

min=0.332, mean=0.339, max=0.343, sum=1.017 (3)

min=0.093, mean=0.105, max=0.113, sum=0.314 (3)

min=0.023, mean=0.023, max=0.025, sum=0.07 (3)

min=0.848, mean=0.945, max=1.053, sum=2.834 (3)

min=0, mean=0.187, max=0.33, sum=0.561 (3)

min=0.369, mean=0.372, max=0.377, sum=1.115 (3)

min=807.577, mean=877.742, max=916.668, sum=2633.225 (3)

min=0.593, mean=0.65, max=0.688, sum=1.95 (3)

min=0.048, mean=0.062, max=0.079, sum=0.185 (3)

min=0.331, mean=0.383, max=0.42, sum=1.148 (3)

min=0.481, mean=0.548, max=0.591, sum=1.644 (3)

min=0.396, mean=0.454, max=0.5, sum=1.362 (3)

min=0.204, mean=0.208, max=0.215, sum=0.624 (3)

min=0.011, mean=0.021, max=0.028, sum=0.062 (3)

min=1.025, mean=1.062, max=1.132, sum=3.185 (3)

min=0.958, mean=1.562, max=1.997, sum=4.687 (3)

min=1601.997, mean=1634.99, max=1693.155, sum=4904.969 (3)

min=5.794, mean=7.077, max=9.031, sum=21.231 (3)

min=0.581, mean=0.625, max=0.647, sum=1.874 (3)

min=0.032, mean=0.037, max=0.044, sum=0.11 (3)

min=0.318, mean=0.357, max=0.38, sum=1.072 (3)

min=0.466, mean=0.512, max=0.538, sum=1.537 (3)

min=0.418, mean=0.473, max=0.5, sum=1.418 (3)

min=0.193, mean=0.202, max=0.211, sum=0.607 (3)

min=0.693, mean=0.729, max=0.782, sum=2.186 (3)

min=5.535, mean=6.91, max=9.504, sum=20.73 (3)

min=0.54, mean=0.559, max=0.572, sum=1.677 (3)

min=0.043, mean=0.047, max=0.055, sum=0.141 (3)

min=0.283, mean=0.3, max=0.315, sum=0.899 (3)

min=0.416, mean=0.438, max=0.455, sum=1.313 (3)

min=0.394, mean=0.427, max=0.45, sum=1.282 (3)

min=0.373, mean=0.569, max=0.667, sum=1.706 (3)

min=0.152, mean=0.174, max=0.195, sum=0.521 (3)

min=0.014, mean=0.02, max=0.025, sum=0.059 (3)

min=0.525, mean=0.533, max=0.548, sum=1.599 (3)

min=5.392, mean=6.771, max=8.33, sum=20.313 (3)

min=0.281, mean=0.294, max=0.309, sum=0.881 (3)

min=0.029, mean=0.031, max=0.033, sum=0.093 (3)

min=0.076, mean=0.078, max=0.081, sum=0.235 (3)

min=0.167, mean=0.179, max=0.197, sum=0.538 (3)

min=0.411, mean=0.418, max=0.429, sum=1.255 (3)

min=0.186, mean=0.202, max=0.217, sum=0.606 (3)

min=0.025, mean=0.027, max=0.031, sum=0.082 (3)

min=0.544, mean=0.56, max=0.583, sum=1.681 (3)

min=8.149, mean=11.007, max=15.597, sum=33.02 (3)

min=0.607, mean=0.672, max=0.708, sum=2.017 (3)

min=0.042, mean=0.059, max=0.072, sum=0.178 (3)

min=0.313, mean=0.39, max=0.434, sum=1.171 (3)

min=0.486, mean=0.553, max=0.589, sum=1.659 (3)

min=0.417, mean=0.472, max=0.5, sum=1.417 (3)

min=0.184, mean=0.192, max=0.197, sum=0.575 (3)

min=0.008, mean=0.013, max=0.02, sum=0.039 (3)

min=5.792, mean=6.729, max=8.434, sum=20.186 (3)

min=0.57, mean=0.61, max=0.642, sum=1.831 (3)

min=0.027, mean=0.028, max=0.03, sum=0.085 (3)

min=0.265, mean=0.296, max=0.321, sum=0.888 (3)

min=0.441, mean=0.497, max=0.537, sum=1.491 (3)

min=0.417, mean=0.441, max=0.469, sum=1.323 (3)

min=0.15, mean=0.181, max=0.213, sum=0.543 (3)

min=0.008, mean=0.011, max=0.014, sum=0.034 (3)

min=5.544, mean=7.144, max=9.065, sum=21.431 (3)

min=0.707, mean=0.709, max=0.712, sum=2.128 (3)

min=0.075, mean=0.076, max=0.077, sum=0.228 (3)

min=0.515, mean=0.529, max=0.539, sum=1.586 (3)

min=0.592, mean=0.595, max=0.6, sum=1.785 (3)

min=0.463, mean=0.488, max=0.5, sum=1.463 (3)

min=0.126, mean=0.144, max=0.169, sum=0.432 (3)

min=0.904, mean=1.508, max=1.941, sum=4.524 (3)

min=1570.772, mean=1600.684, max=1660.485, sum=4802.051 (3)