The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

BBQ (ambiguous): Metric of [Parrish et al. (2022)](https://aclanthology.org/2022.findings-acl.165/) for BBQ on ambiguous examples.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

BBQ (unambiguous): Metric of [Parrish et al. (2022)](https://aclanthology.org/2022.findings-acl.165/) for BBQ on unambiguous examples.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

# eval: Number of evaluation instances.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

# train: Number of training instances (e.g., in-context examples).

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

# prompt tokens: Number of tokens in the prompt.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

# output tokens: Actual number of output tokens.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

bbq_subject:all

min=0.342, mean=0.378, max=0.415, sum=1.133 (3)

min=-0.248, mean=-0.112, max=-0.03, sum=-0.335 (3)

min=-0.352, mean=-0.265, max=-0.165, sum=-0.796 (3)

min=0.413, mean=0.421, max=0.437, sum=1.264 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=298.847, mean=314.18, max=343.847, sum=942.541 (3)

min=0.329, mean=0.352, max=0.365, sum=1.055 (3)

min=-0.227, mean=-0.133, max=-0.023, sum=-0.398 (3)

min=-0.273, mean=-0.225, max=-0.161, sum=-0.675 (3)

min=0.344, mean=0.35, max=0.361, sum=1.05 (3)

min=0.359, mean=0.361, max=0.363, sum=1.083 (3)

min=-0.062, mean=-0.047, max=-0.031, sum=-0.142 (3)

min=-0.184, mean=-0.164, max=-0.153, sum=-0.493 (3)

min=0.377, mean=0.383, max=0.393, sum=1.148 (3)

min=0.439, mean=0.476, max=0.525, sum=1.428 (3)

min=-0.117, mean=-0.067, max=-0.033, sum=-0.2 (3)

min=-0.23, mean=-0.212, max=-0.185, sum=-0.636 (3)

min=0.521, mean=0.55, max=0.573, sum=1.651 (3)

min=0.003, mean=0.039, max=0.061, sum=0.116 (3)

min=-0.176, mean=-0.146, max=-0.119, sum=-0.438 (3)

min=0.444, mean=0.494, max=0.52, sum=1.482 (3)

min=-0.128, mean=-0.069, max=-0.034, sum=-0.206 (3)

min=-0.288, mean=-0.202, max=-0.099, sum=-0.606 (3)

min=0.314, mean=0.347, max=0.4, sum=1.041 (3)

min=-0.097, mean=-0.06, max=-0.041, sum=-0.181 (3)

min=-0.163, mean=-0.124, max=-0.093, sum=-0.373 (3)

min=0.305, mean=0.333, max=0.377, sum=0.998 (3)

min=-0.263, mean=-0.198, max=-0.08, sum=-0.594 (3)

min=-0.337, mean=-0.237, max=-0.165, sum=-0.71 (3)

min=398.965, mean=424.298, max=460.965, sum=1272.895 (3)

min=0.342, mean=0.355, max=0.37, sum=1.065 (3)

min=-0.244, mean=-0.149, max=-0.037, sum=-0.447 (3)

min=-0.299, mean=-0.257, max=-0.176, sum=-0.77 (3)

min=0.393, mean=0.411, max=0.441, sum=1.233 (3)

min=-0.148, mean=-0.108, max=-0.054, sum=-0.324 (3)

min=-0.207, mean=-0.161, max=-0.117, sum=-0.483 (3)

min=0.531, mean=0.551, max=0.574, sum=1.653 (3)

min=-0.037, mean=-0.002, max=0.02, sum=-0.005 (3)

min=-0.159, mean=-0.141, max=-0.124, sum=-0.424 (3)

min=0.593, mean=0.601, max=0.613, sum=1.804 (3)

min=401.437, mean=424.437, max=460.437, sum=1273.311 (3)

min=0.361, mean=0.375, max=0.39, sum=1.124 (3)

min=-0.101, mean=-0.071, max=-0.047, sum=-0.214 (3)

min=-0.235, mean=-0.222, max=-0.201, sum=-0.667 (3)

min=0.102, mean=0.124, max=0.139, sum=0.372 (3)

min=370.628, mean=396.628, max=434.628, sum=1189.884 (3)

min=0.488, mean=0.491, max=0.495, sum=1.474 (3)

min=0.014, mean=0.02, max=0.024, sum=0.059 (3)

min=-0.182, mean=-0.174, max=-0.161, sum=-0.522 (3)

min=0.145, mean=0.145, max=0.146, sum=0.436 (3)

min=411.763, mean=432.096, max=470.763, sum=1296.289 (3)

min=0.315, mean=0.345, max=0.362, sum=1.034 (3)

min=-0.048, mean=-0.03, max=-0.012, sum=-0.09 (3)

min=-0.189, mean=-0.129, max=-0.07, sum=-0.388 (3)

min=0.484, mean=0.494, max=0.507, sum=1.481 (3)

min=404.745, mean=428.412, max=463.745, sum=1285.235 (3)

min=0.318, mean=0.328, max=0.342, sum=0.983 (3)

min=-0.087, mean=-0.061, max=-0.046, sum=-0.182 (3)

min=-0.134, mean=-0.089, max=-0.062, sum=-0.266 (3)

min=0.3, mean=0.306, max=0.315, sum=0.919 (3)

min=0.308, mean=0.344, max=0.362, sum=1.031 (3)

min=-0.066, mean=-0.043, max=-0.031, sum=-0.128 (3)

min=-0.188, mean=-0.167, max=-0.157, sum=-0.501 (3)

min=0.271, mean=0.276, max=0.283, sum=0.829 (3)

min=0.353, mean=0.363, max=0.374, sum=1.088 (3)

min=-0.049, mean=-0.029, max=-0.007, sum=-0.087 (3)

min=-0.151, mean=-0.112, max=-0.075, sum=-0.337 (3)

min=0.269, mean=0.273, max=0.278, sum=0.818 (3)

min=0.372, mean=0.397, max=0.416, sum=1.192 (3)

min=-0.153, mean=-0.112, max=-0.069, sum=-0.335 (3)

min=-0.212, mean=-0.202, max=-0.186, sum=-0.605 (3)

min=0.342, mean=0.359, max=0.369, sum=1.076 (3)

min=-0.061, mean=-0.057, max=-0.049, sum=-0.17 (3)

min=-0.217, mean=-0.176, max=-0.152, sum=-0.528 (3)

min=0.288, mean=0.302, max=0.309, sum=0.905 (3)

min=-0.046, mean=-0.034, max=-0.027, sum=-0.102 (3)

min=-0.098, mean=-0.078, max=-0.064, sum=-0.234 (3)

min=0.454, mean=0.463, max=0.473, sum=1.39 (3)

min=-0.032, mean=-0.018, max=0, sum=-0.053 (3)

min=-0.186, mean=-0.179, max=-0.168, sum=-0.537 (3)

min=0.303, mean=0.307, max=0.316, sum=0.922 (3)

min=-0.158, mean=-0.054, max=0.021, sum=-0.161 (3)

min=-0.117, mean=-0.1, max=-0.082, sum=-0.3 (3)

min=0.053, mean=0.069, max=0.101, sum=0.208 (3)

min=0.29, mean=0.316, max=0.343, sum=0.949 (3)

min=-0.152, mean=-0.075, max=-0.031, sum=-0.226 (3)

min=-0.158, mean=-0.145, max=-0.132, sum=-0.434 (3)

min=0.127, mean=0.226, max=0.275, sum=0.678 (3)

min=402.456, mean=426.123, max=461.456, sum=1278.368 (3)

min=0.279, mean=0.312, max=0.34, sum=0.935 (3)

min=-0.21, mean=-0.174, max=-0.107, sum=-0.522 (3)

min=-0.361, mean=-0.343, max=-0.324, sum=-1.029 (3)

min=0.175, mean=0.191, max=0.218, sum=0.573 (3)

min=4.915, mean=4.972, max=5, sum=14.915 (3)

min=411.763, mean=429.546, max=463.113, sum=1288.639 (3)

min=0.3, mean=0.321, max=0.35, sum=0.964 (3)

min=-0.046, mean=-0.036, max=-0.025, sum=-0.108 (3)

min=-0.173, mean=-0.137, max=-0.099, sum=-0.41 (3)

min=0.179, mean=0.179, max=0.179, sum=0.538 (3)

min=4.899, mean=4.966, max=5, sum=14.899 (3)

min=415.763, mean=433.066, max=465.673, sum=1299.199 (3)

min=0.339, mean=0.347, max=0.351, sum=1.041 (3)

min=-0.23, mean=-0.187, max=-0.104, sum=-0.562 (3)

min=-0.33, mean=-0.254, max=-0.169, sum=-0.763 (3)

min=0.266, mean=0.285, max=0.296, sum=0.856 (3)

min=0.342, mean=0.355, max=0.381, sum=1.066 (3)

min=-0.249, mean=-0.179, max=-0.099, sum=-0.536 (3)

min=-0.36, mean=-0.329, max=-0.295, sum=-0.987 (3)

min=0.042, mean=0.043, max=0.043, sum=0.128 (3)

min=0.452, mean=0.479, max=0.499, sum=1.438 (3)

min=-0.117, mean=-0.034, max=0.036, sum=-0.103 (3)

min=-0.17, mean=-0.157, max=-0.139, sum=-0.472 (3)

min=0.312, mean=0.337, max=0.366, sum=1.01 (3)

min=-0.079, mean=-0.058, max=-0.045, sum=-0.174 (3)

min=-0.19, mean=-0.137, max=-0.076, sum=-0.411 (3)

min=0.361, mean=0.39, max=0.422, sum=1.17 (3)

min=-0.105, mean=-0.048, max=-0.015, sum=-0.144 (3)

min=-0.222, mean=-0.184, max=-0.149, sum=-0.552 (3)

min=0.218, mean=0.22, max=0.223, sum=0.66 (3)

min=0.335, mean=0.362, max=0.391, sum=1.087 (3)

min=-0.094, mean=-0.053, max=-0.016, sum=-0.158 (3)

min=-0.223, mean=-0.17, max=-0.124, sum=-0.509 (3)

min=0.092, mean=0.092, max=0.093, sum=0.277 (3)

min=0.291, mean=0.319, max=0.346, sum=0.957 (3)

min=-0.223, mean=-0.113, max=-0.013, sum=-0.339 (3)

min=-0.32, mean=-0.275, max=-0.19, sum=-0.825 (3)

min=0.12, mean=0.12, max=0.121, sum=0.36 (3)

min=0.313, mean=0.322, max=0.327, sum=0.965 (3)

min=-0.252, mean=-0.182, max=-0.046, sum=-0.546 (3)

min=-0.348, mean=-0.258, max=-0.083, sum=-0.774 (3)

min=0.141, mean=0.141, max=0.141, sum=0.423 (3)

min=0.776, mean=0.862, max=0.919, sum=2.585 (3)

min=-0.189, mean=-0.179, max=-0.171, sum=-0.538 (3)

min=0.855, mean=0.881, max=0.904, sum=2.642 (3)

min=0.026, mean=0.033, max=0.045, sum=0.099 (3)

min=-0.179, mean=-0.171, max=-0.166, sum=-0.512 (3)

min=0.21, mean=0.213, max=0.217, sum=0.639 (3)

min=0.311, mean=0.342, max=0.395, sum=1.026 (3)

min=-0.156, mean=-0.077, max=-0.017, sum=-0.231 (3)

min=-0.178, mean=-0.153, max=-0.122, sum=-0.46 (3)

min=0.13, mean=0.131, max=0.132, sum=0.392 (3)

min=0.344, mean=0.359, max=0.372, sum=1.077 (3)

min=-0.099, mean=-0.077, max=-0.039, sum=-0.23 (3)

min=-0.217, mean=-0.197, max=-0.173, sum=-0.59 (3)

min=0.131, mean=0.132, max=0.133, sum=0.396 (3)

min=0.333, mean=0.359, max=0.377, sum=1.077 (3)

min=-0.142, mean=-0.093, max=-0.062, sum=-0.279 (3)

min=-0.19, mean=-0.118, max=-0.031, sum=-0.355 (3)

min=0.087, mean=0.088, max=0.089, sum=0.263 (3)

min=0.882, mean=0.9, max=0.934, sum=2.699 (3)

min=-0.053, mean=-0.037, max=-0.013, sum=-0.112 (3)

min=-0.187, mean=-0.183, max=-0.179, sum=-0.55 (3)

min=381.014, mean=403.681, max=440.014, sum=1211.042 (3)

min=1.001, mean=1.002, max=1.002, sum=3.005 (3)

min=0.311, mean=0.311, max=0.311, sum=0.311 (1)

min=-0.055, mean=-0.055, max=-0.055, sum=-0.055 (1)

min=-0.078, mean=-0.078, max=-0.078, sum=-0.078 (1)

min=1000, mean=1000, max=1000, sum=1000 (1)

min=402.456, mean=402.456, max=402.456, sum=402.456 (1)

min=0.284, mean=0.316, max=0.333, sum=0.948 (3)

min=-0.182, mean=-0.069, max=-0.002, sum=-0.208 (3)

min=-0.266, mean=-0.19, max=-0.142, sum=-0.571 (3)

min=0.431, mean=0.482, max=0.509, sum=1.447 (3)

min=364.702, mean=388.369, max=424.702, sum=1165.106 (3)

min=0.353, mean=0.37, max=0.388, sum=1.111 (3)

min=-0.052, mean=-0.012, max=0.02, sum=-0.035 (3)

min=-0.175, mean=-0.154, max=-0.118, sum=-0.461 (3)

min=0.999, mean=1.0, max=1, sum=2.999 (3)

min=0.745, mean=0.781, max=0.805, sum=2.344 (3)

min=0.039, mean=0.04, max=0.042, sum=0.12 (3)

min=-0.182, mean=-0.174, max=-0.169, sum=-0.523 (3)

min=0.941, mean=0.958, max=0.967, sum=2.874 (3)

min=0.327, mean=0.328, max=0.33, sum=0.984 (3)

min=-0.251, mean=-0.25, max=-0.247, sum=-0.75 (3)

min=-0.357, mean=-0.356, max=-0.353, sum=-1.068 (3)

min=0.091, mean=0.093, max=0.096, sum=0.279 (3)

min=403.995, mean=429.662, max=466.995, sum=1288.985 (3)