The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

quac

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

F1: Average F1 score in terms of word overlap between the model output and correct reference.

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

F1: Average F1 score in terms of word overlap between the model output and correct reference.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

# eval: Number of evaluation instances.

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

# train: Number of training instances (e.g., in-context examples).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

# prompt tokens: Number of tokens in the prompt.

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

# output tokens: Actual number of output tokens.

The QuAC benchmark for question answering in the context of dialogues [(Choi et al., 2018)](https://aclanthology.org/D18-1241/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

quac_

min=0.348, mean=0.358, max=0.372, sum=1.075 (3)

min=0.042, mean=0.043, max=0.045, sum=0.13 (3)

min=0.216, mean=0.222, max=0.232, sum=0.667 (3)

min=0.263, mean=0.268, max=0.275, sum=0.805 (3)

min=0.594, mean=0.604, max=0.613, sum=1.811 (3)

min=0.417, mean=0.42, max=0.425, sum=1.26 (3)

min=0.287, mean=0.329, max=0.362, sum=0.988 (3)

min=0.231, mean=0.242, max=0.257, sum=0.725 (3)

min=0.002, mean=0.003, max=0.004, sum=0.009 (3)

min=1.898, mean=2.064, max=2.149, sum=6.193 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=1.788, mean=1.829, max=1.88, sum=5.486 (3)

min=0.001, mean=0.001, max=0.001, sum=0.003 (3)

min=1645.856, mean=1698.711, max=1730.814, sum=5096.134 (3)

min=22.621, mean=26.784, max=29.261, sum=80.351 (3)

min=0.322, mean=0.328, max=0.336, sum=0.983 (3)

min=0.016, mean=0.024, max=0.033, sum=0.073 (3)

min=0.186, mean=0.197, max=0.209, sum=0.591 (3)

min=0.227, mean=0.241, max=0.256, sum=0.722 (3)

min=0.632, mean=0.647, max=0.667, sum=1.942 (3)

min=0.407, mean=0.428, max=0.446, sum=1.284 (3)

min=0.226, mean=0.3, max=0.351, sum=0.9 (3)

min=0.235, mean=0.249, max=0.271, sum=0.748 (3)

min=0.002, mean=0.003, max=0.004, sum=0.008 (3)

min=1.105, mean=1.16, max=1.191, sum=3.48 (3)

min=23.833, mean=27.642, max=30.067, sum=82.927 (3)

min=0.355, mean=0.362, max=0.372, sum=1.087 (3)

min=0.019, mean=0.036, max=0.06, sum=0.107 (3)

min=0.215, mean=0.219, max=0.227, sum=0.658 (3)

min=0.266, mean=0.274, max=0.282, sum=0.821 (3)

min=0.58, mean=0.6, max=0.639, sum=1.799 (3)

min=0.415, mean=0.428, max=0.44, sum=1.283 (3)

min=0.298, mean=0.34, max=0.378, sum=1.019 (3)

min=0.237, mean=0.242, max=0.25, sum=0.727 (3)

min=0.004, mean=0.004, max=0.004, sum=0.012 (3)

min=1.302, mean=1.413, max=1.478, sum=4.24 (3)

min=22.154, mean=27.786, max=31.692, sum=83.357 (3)

min=0.375, mean=0.392, max=0.411, sum=1.177 (3)

min=0.031, mean=0.04, max=0.051, sum=0.121 (3)

min=0.232, mean=0.251, max=0.261, sum=0.752 (3)

min=0.297, mean=0.308, max=0.319, sum=0.923 (3)

min=0.583, mean=0.628, max=0.66, sum=1.884 (3)

min=0.396, mean=0.411, max=0.426, sum=1.232 (3)

min=0.302, mean=0.327, max=0.359, sum=0.981 (3)

min=0.198, mean=0.225, max=0.241, sum=0.676 (3)

min=0.003, mean=0.003, max=0.004, sum=0.01 (3)

min=19.318, mean=23.053, max=25.3, sum=69.158 (3)

min=0.411, mean=0.421, max=0.428, sum=1.264 (3)

min=0.044, mean=0.056, max=0.062, sum=0.168 (3)

min=0.279, mean=0.292, max=0.299, sum=0.875 (3)

min=0.313, mean=0.327, max=0.336, sum=0.981 (3)

min=0.587, mean=0.618, max=0.646, sum=1.855 (3)

min=0.423, mean=0.432, max=0.441, sum=1.297 (3)

min=0.261, mean=0.282, max=0.3, sum=0.846 (3)

min=0.221, mean=0.24, max=0.265, sum=0.721 (3)

min=0, mean=0.0, max=0.001, sum=0.001 (3)

min=4.999, mean=5.0, max=5, sum=14.999 (3)

min=3596.32, mean=4027.779, max=4577.698, sum=12083.338 (3)

min=22.825, mean=24.163, max=26.151, sum=72.49 (3)

min=0.38, mean=0.393, max=0.4, sum=1.178 (3)

min=0.033, mean=0.045, max=0.055, sum=0.136 (3)

min=0.244, mean=0.247, max=0.25, sum=0.741 (3)

min=0.296, mean=0.307, max=0.32, sum=0.921 (3)

min=0.571, mean=0.603, max=0.625, sum=1.808 (3)

min=0.444, mean=0.454, max=0.463, sum=1.362 (3)

min=0.265, mean=0.339, max=0.381, sum=1.016 (3)

min=0.197, mean=0.236, max=0.26, sum=0.707 (3)

min=0.002, mean=0.002, max=0.002, sum=0.006 (3)

min=1.779, mean=1.821, max=1.873, sum=5.462 (3)

min=1650.921, mean=1703.333, max=1734.801, sum=5110 (3)

min=17.917, mean=20.427, max=22.478, sum=61.282 (3)

min=0.354, mean=0.367, max=0.378, sum=1.1 (3)

min=0.047, mean=0.054, max=0.062, sum=0.162 (3)

min=0.217, mean=0.226, max=0.236, sum=0.677 (3)

min=0.263, mean=0.271, max=0.279, sum=0.812 (3)

min=0.601, mean=0.607, max=0.615, sum=1.822 (3)

min=0.389, mean=0.424, max=0.46, sum=1.271 (3)

min=0.272, mean=0.329, max=0.371, sum=0.987 (3)

min=0.223, mean=0.242, max=0.261, sum=0.727 (3)

min=0.002, mean=0.004, max=0.005, sum=0.012 (3)

min=20.062, mean=24.427, max=28.09, sum=73.282 (3)

min=0.317, mean=0.334, max=0.362, sum=1.003 (3)

min=0.068, mean=0.098, max=0.131, sum=0.295 (3)

min=0.166, mean=0.185, max=0.212, sum=0.556 (3)

min=0.251, mean=0.266, max=0.284, sum=0.799 (3)

min=0.641, mean=0.658, max=0.667, sum=1.974 (3)

min=0.401, mean=0.417, max=0.432, sum=1.251 (3)

min=0.258, mean=0.32, max=0.377, sum=0.96 (3)

min=0.193, mean=0.203, max=0.212, sum=0.61 (3)

min=0.001, mean=0.002, max=0.003, sum=0.006 (3)

min=0.84, mean=0.909, max=0.991, sum=2.727 (3)

min=0.029, mean=0.033, max=0.037, sum=0.098 (3)

min=1596.904, mean=1641.256, max=1672.92, sum=4923.768 (3)

min=18.527, mean=23.472, max=28.795, sum=70.415 (3)

min=0.34, mean=0.349, max=0.363, sum=1.047 (3)

min=0.081, mean=0.096, max=0.116, sum=0.287 (3)

min=0.188, mean=0.193, max=0.201, sum=0.578 (3)

min=0.268, mean=0.277, max=0.295, sum=0.832 (3)

min=0.59, mean=0.612, max=0.636, sum=1.837 (3)

min=0.382, mean=0.403, max=0.421, sum=1.208 (3)

min=0.202, mean=0.24, max=0.259, sum=0.719 (3)

min=0.194, mean=0.2, max=0.205, sum=0.601 (3)

min=20.299, mean=21.144, max=22.408, sum=63.432 (3)

min=0.364, mean=0.37, max=0.378, sum=1.111 (3)

min=0.054, mean=0.058, max=0.061, sum=0.175 (3)

min=0.221, mean=0.233, max=0.24, sum=0.699 (3)

min=0.28, mean=0.288, max=0.3, sum=0.865 (3)

min=0.571, mean=0.598, max=0.615, sum=1.794 (3)

min=0.408, mean=0.412, max=0.415, sum=1.236 (3)

min=0.269, mean=0.305, max=0.351, sum=0.914 (3)

min=0.227, mean=0.232, max=0.235, sum=0.696 (3)

min=0.002, mean=0.002, max=0.003, sum=0.007 (3)

min=22.638, mean=26.241, max=28.094, sum=78.723 (3)

min=0.41, mean=0.431, max=0.443, sum=1.294 (3)

min=0.034, mean=0.039, max=0.048, sum=0.116 (3)

min=0.303, mean=0.313, max=0.324, sum=0.938 (3)

min=0.338, mean=0.356, max=0.365, sum=1.067 (3)

min=0.604, mean=0.609, max=0.614, sum=1.827 (3)

min=0.405, mean=0.419, max=0.441, sum=1.257 (3)

min=0.283, mean=0.321, max=0.341, sum=0.964 (3)

min=0.246, mean=0.248, max=0.249, sum=0.743 (3)

min=0.001, mean=0.002, max=0.002, sum=0.005 (3)

min=3.472, mean=3.694, max=4.123, sum=11.082 (3)

min=4676.788, mean=5199.788, max=5842.788, sum=15599.364 (3)

min=32.106, mean=35.484, max=40.222, sum=106.452 (3)

min=0.342, mean=0.361, max=0.375, sum=1.082 (3)

min=0.103, mean=0.122, max=0.142, sum=0.367 (3)

min=0.229, mean=0.234, max=0.24, sum=0.701 (3)

min=0.265, mean=0.273, max=0.289, sum=0.82 (3)

min=0.604, mean=0.631, max=0.647, sum=1.894 (3)

min=0.388, mean=0.396, max=0.408, sum=1.189 (3)

min=0.35, mean=0.365, max=0.381, sum=1.094 (3)

min=0.235, mean=0.244, max=0.26, sum=0.732 (3)

min=0, mean=0.001, max=0.002, sum=0.003 (3)

min=5.124, mean=5.306, max=5.436, sum=15.919 (3)

min=0.855, mean=0.944, max=1.07, sum=2.832 (3)

min=0.017, mean=0.017, max=0.017, sum=0.051 (3)

min=1614.308, mean=1639.494, max=1673.303, sum=4918.482 (3)

min=86.351, mean=90.164, max=93.357, sum=270.491 (3)

min=0.121, mean=0.121, max=0.121, sum=0.362 (3)

min=0.001, mean=0.001, max=0.001, sum=0.002 (3)

min=0.071, mean=0.071, max=0.071, sum=0.212 (3)

min=0.067, mean=0.067, max=0.067, sum=0.201 (3)

min=0.667, mean=0.667, max=0.667, sum=2 (3)

min=0.428, mean=0.428, max=0.428, sum=1.284 (3)

min=0.436, mean=0.436, max=0.436, sum=1.308 (3)

min=0.291, mean=0.291, max=0.291, sum=0.872 (3)

min=1.239, mean=1.239, max=1.239, sum=3.716 (3)

min=0.985, mean=0.985, max=0.985, sum=2.955 (3)

min=823.365, mean=823.365, max=823.365, sum=2470.095 (3)

min=0.355, mean=0.361, max=0.365, sum=1.082 (3)

min=0.066, mean=0.067, max=0.07, sum=0.201 (3)

min=0.214, mean=0.215, max=0.216, sum=0.646 (3)

min=0.274, mean=0.281, max=0.287, sum=0.844 (3)

min=0.556, mean=0.582, max=0.6, sum=1.745 (3)

min=0.43, mean=0.438, max=0.449, sum=1.315 (3)

min=0.333, mean=0.344, max=0.355, sum=1.033 (3)

min=0.223, mean=0.23, max=0.237, sum=0.691 (3)

min=2.057, mean=2.089, max=2.151, sum=6.267 (3)

min=0.797, mean=0.881, max=0.969, sum=2.644 (3)

min=0.02, mean=0.02, max=0.02, sum=0.06 (3)

min=1600.292, mean=1639.784, max=1661.675, sum=4919.353 (3)

min=31.783, mean=32.717, max=34.585, sum=98.152 (3)

min=0.335, mean=0.338, max=0.343, sum=1.015 (3)

min=0.03, mean=0.033, max=0.036, sum=0.099 (3)

min=0.197, mean=0.204, max=0.211, sum=0.613 (3)

min=0.251, mean=0.256, max=0.259, sum=0.768 (3)

min=0.43, mean=0.441, max=0.46, sum=1.322 (3)

min=0.306, mean=0.338, max=0.358, sum=1.015 (3)

min=0.234, mean=0.238, max=0.243, sum=0.714 (3)

min=1.189, mean=1.262, max=1.309, sum=3.785 (3)

min=26.693, mean=30.036, max=32.515, sum=90.109 (3)

min=0.273, mean=0.279, max=0.287, sum=0.838 (3)

min=0.042, mean=0.048, max=0.061, sum=0.145 (3)

min=0.12, mean=0.144, max=0.157, sum=0.432 (3)

min=0.186, mean=0.198, max=0.207, sum=0.593 (3)

min=0.667, mean=0.667, max=0.667, sum=2.0 (3)

min=0.383, mean=0.412, max=0.431, sum=1.237 (3)

min=0.303, mean=0.357, max=0.392, sum=1.072 (3)

min=0.233, mean=0.262, max=0.276, sum=0.786 (3)

min=0.664, mean=0.735, max=0.771, sum=2.206 (3)

min=17.39, mean=23.531, max=27.056, sum=70.593 (3)