Vary the adapation strategy for multiple-choice questions.

ablation_multiple_choice

How many models this model outperform on average (over columns).

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The OpenbookQA benchmark for commonsense-intensive open book question answering [(Mihaylov et al., 2018)](https://aclanthology.org/D18-1260/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The TruthfulQA benchmarking for measuring model truthfulness and commonsense knowledge in question answering [(Lin et al., 2022)](https://aclanthology.org/2022.acl-long.229/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context [(Parrish et al., 2022)](https://aclanthology.org/2022.findings-acl.165/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

accuracy

min=0.539, mean=0.543, max=0.549, sum=3.26 (6)

min=0.672, mean=0.684, max=0.69, sum=4.104 (6)

min=0.298, mean=0.361, max=0.408, sum=2.168 (6)

min=0.25, mean=0.481, max=0.78, sum=14.439 (30)

min=0.622, mean=0.624, max=0.628, sum=3.746 (6)

min=0.2, mean=0.213, max=0.23, sum=1.278 (6)

min=0.531, mean=0.551, max=0.574, sum=3.306 (6)

min=0.593, mean=0.593, max=0.593, sum=1.186 (2)

min=0.558, mean=0.558, max=0.558, sum=1.116 (2)

min=0.332, mean=0.332, max=0.332, sum=0.664 (2)

min=0.298, mean=0.408, max=0.56, sum=4.076 (10)

min=0.493, mean=0.529, max=0.575, sum=4.23 (8)

min=0.569, mean=0.569, max=0.569, sum=1.137 (2)

min=0.191, mean=0.191, max=0.191, sum=0.383 (2)

min=0.259, mean=0.259, max=0.259, sum=0.518 (2)

min=0.807, mean=0.807, max=0.807, sum=1.614 (2)

min=0.446, mean=0.446, max=0.446, sum=0.892 (2)

min=0.257, mean=0.257, max=0.257, sum=0.514 (2)

min=0.289, mean=0.37, max=0.46, sum=3.699 (10)

min=0.772, mean=0.829, max=0.937, sum=6.634 (8)

min=0.544, mean=0.544, max=0.544, sum=1.088 (2)

min=0.222, mean=0.222, max=0.222, sum=0.443 (2)

min=0.279, mean=0.279, max=0.279, sum=0.558 (2)

min=0.278, mean=0.289, max=0.299, sum=1.736 (6)

min=0.342, mean=0.351, max=0.364, sum=2.104 (6)

min=0.197, mean=0.204, max=0.211, sum=1.226 (6)

min=0.19, mean=0.299, max=0.42, sum=8.961 (30)

min=0.112, mean=0.249, max=0.348, sum=5.976 (24)

min=0.515, mean=0.543, max=0.558, sum=3.256 (6)

min=0.2, mean=0.209, max=0.217, sum=1.252 (6)

min=0.361, mean=0.375, max=0.39, sum=2.248 (6)

min=0.548, mean=0.548, max=0.548, sum=1.096 (2)

min=0.534, mean=0.534, max=0.534, sum=1.068 (2)

min=0.358, mean=0.358, max=0.358, sum=0.716 (2)

min=0.29, mean=0.411, max=0.58, sum=4.112 (10)

min=0.468, mean=0.527, max=0.657, sum=4.214 (8)

min=0.566, mean=0.566, max=0.566, sum=1.133 (2)

min=0.243, mean=0.243, max=0.243, sum=0.487 (2)

min=0.263, mean=0.263, max=0.263, sum=0.526 (2)

min=0.744, mean=0.744, max=0.744, sum=1.488 (2)

min=0.448, mean=0.448, max=0.448, sum=0.896 (2)

min=0.242, mean=0.242, max=0.242, sum=0.483 (2)

min=0.28, mean=0.345, max=0.43, sum=3.454 (10)

min=0.743, mean=0.819, max=0.929, sum=6.552 (8)

min=0.579, mean=0.579, max=0.579, sum=1.157 (2)

min=0.253, mean=0.265, max=0.271, sum=1.588 (6)

min=0.256, mean=0.282, max=0.296, sum=1.692 (6)

min=0.187, mean=0.196, max=0.213, sum=1.174 (6)

min=0.14, mean=0.249, max=0.3, sum=7.456 (30)

min=0.058, mean=0.242, max=0.469, sum=5.808 (24)

min=0.47, mean=0.479, max=0.491, sum=2.871 (6)

min=0.143, mean=0.175, max=0.2, sum=1.052 (6)

min=0.303, mean=0.307, max=0.316, sum=1.844 (6)

min=0.49, mean=0.49, max=0.49, sum=0.98 (2)

min=0.514, mean=0.514, max=0.514, sum=1.028 (2)

min=0.396, mean=0.396, max=0.396, sum=0.792 (2)

min=0.22, mean=0.35, max=0.48, sum=3.499 (10)

min=0.416, mean=0.494, max=0.552, sum=3.956 (8)

min=0.558, mean=0.558, max=0.558, sum=1.117 (2)

min=0.239, mean=0.239, max=0.239, sum=0.478 (2)

min=0.27, mean=0.27, max=0.27, sum=0.54 (2)

min=0.663, mean=0.663, max=0.663, sum=1.326 (2)

min=0.42, mean=0.42, max=0.42, sum=0.84 (2)

min=0.228, mean=0.328, max=0.43, sum=3.276 (10)

min=0.725, mean=0.834, max=0.938, sum=6.672 (8)

min=0.509, mean=0.509, max=0.509, sum=1.018 (2)

min=0.204, mean=0.204, max=0.204, sum=0.409 (2)

min=0.266, mean=0.271, max=0.28, sum=1.628 (6)

min=0.274, mean=0.28, max=0.284, sum=1.68 (6)

min=0.205, mean=0.213, max=0.219, sum=1.278 (6)

min=0.21, mean=0.276, max=0.351, sum=8.292 (30)

min=0, mean=0.009, max=0.035, sum=0.222 (24)

min=0.503, mean=0.515, max=0.53, sum=3.088 (6)

min=0.152, mean=0.191, max=0.217, sum=1.148 (6)

min=0.29, mean=0.316, max=0.343, sum=1.898 (6)

min=0.524, mean=0.524, max=0.524, sum=1.048 (2)

min=0.37, mean=0.37, max=0.37, sum=0.74 (2)

min=0.263, mean=0.371, max=0.5, sum=3.706 (10)

min=0.423, mean=0.541, max=0.705, sum=4.332 (8)

min=0.526, mean=0.526, max=0.526, sum=1.051 (2)

min=0.226, mean=0.226, max=0.226, sum=0.452 (2)

min=0.253, mean=0.253, max=0.253, sum=0.506 (2)

min=0.718, mean=0.718, max=0.718, sum=1.436 (2)

min=0.24, mean=0.24, max=0.24, sum=0.48 (2)

min=0.254, mean=0.361, max=0.43, sum=3.609 (10)

min=0.742, mean=0.839, max=0.951, sum=6.708 (8)

min=0.511, mean=0.511, max=0.511, sum=1.022 (2)

min=0.235, mean=0.235, max=0.235, sum=0.47 (2)

min=0.295, mean=0.305, max=0.311, sum=1.832 (6)

min=0.332, mean=0.341, max=0.36, sum=2.048 (6)

min=0.228, mean=0.251, max=0.269, sum=1.505 (6)

min=0.21, mean=0.318, max=0.48, sum=9.551 (30)

min=0.513, mean=0.532, max=0.546, sum=3.19 (6)

min=0.191, mean=0.22, max=0.243, sum=1.322 (6)

min=0.339, mean=0.347, max=0.351, sum=2.082 (6)

min=0.586, mean=0.586, max=0.586, sum=1.172 (2)

min=0.405, mean=0.405, max=0.405, sum=0.81 (2)

min=0.19, mean=0.35, max=0.57, sum=3.499 (10)

min=0.266, mean=0.47, max=0.635, sum=3.762 (8)

min=0.556, mean=0.556, max=0.556, sum=1.112 (2)

min=0.209, mean=0.209, max=0.209, sum=0.417 (2)

min=0.273, mean=0.273, max=0.273, sum=0.546 (2)

min=0.791, mean=0.791, max=0.791, sum=1.582 (2)

min=0.292, mean=0.292, max=0.292, sum=0.584 (2)

min=0.254, mean=0.353, max=0.43, sum=3.529 (10)

min=0.747, mean=0.831, max=0.892, sum=6.644 (8)

min=0.161, mean=0.161, max=0.161, sum=0.322 (2)

min=0.269, mean=0.269, max=0.269, sum=0.538 (2)

min=0.242, mean=0.253, max=0.265, sum=1.52 (6)

min=0.236, mean=0.257, max=0.278, sum=1.54 (6)

min=0.185, mean=0.199, max=0.22, sum=1.196 (6)

min=0.2, mean=0.276, max=0.37, sum=8.281 (30)

min=0, mean=0.001, max=0.002, sum=0.012 (24)

min=0.521, mean=0.527, max=0.538, sum=3.162 (6)

min=0.165, mean=0.175, max=0.191, sum=1.052 (6)

min=0.342, mean=0.355, max=0.381, sum=2.132 (6)

min=0.525, mean=0.525, max=0.525, sum=1.05 (2)

min=0.385, mean=0.385, max=0.385, sum=0.771 (2)

min=0.24, mean=0.36, max=0.51, sum=3.599 (10)

min=0.249, mean=0.459, max=0.605, sum=3.674 (8)

min=0.17, mean=0.17, max=0.17, sum=0.339 (2)

min=0.265, mean=0.265, max=0.265, sum=0.53 (2)

min=0.745, mean=0.745, max=0.745, sum=1.49 (2)

min=0.41, mean=0.41, max=0.41, sum=0.82 (2)

min=0.275, mean=0.275, max=0.275, sum=0.55 (2)

min=0.237, mean=0.337, max=0.43, sum=3.374 (10)

min=0.741, mean=0.827, max=0.882, sum=6.614 (8)

min=0.573, mean=0.573, max=0.573, sum=1.145 (2)

min=0.157, mean=0.157, max=0.157, sum=0.313 (2)

min=0.274, mean=0.274, max=0.274, sum=0.548 (2)

Accuracy

The HellaSwag benchmark for commonsense reasoning in question answering [(Zellers et al., 2019)](https://aclanthology.org/P19-1472/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The OpenbookQA benchmark for commonsense-intensive open book question answering [(Mihaylov et al., 2018)](https://aclanthology.org/D18-1260/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The TruthfulQA benchmarking for measuring model truthfulness and commonsense knowledge in question answering [(Lin et al., 2022)](https://aclanthology.org/2022.acl-long.229/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

calibration

min=0.037, mean=0.052, max=0.071, sum=0.312 (6)

min=0.13, mean=0.151, max=0.163, sum=0.906 (6)

min=0.091, mean=0.152, max=0.196, sum=0.912 (6)

min=0.063, mean=0.144, max=0.262, sum=4.33 (30)

min=0.348, mean=0.348, max=0.348, sum=0.696 (2)

min=0.244, mean=0.244, max=0.244, sum=0.487 (2)

min=0.454, mean=0.454, max=0.454, sum=0.908 (2)

min=0.282, mean=0.374, max=0.532, sum=3.737 (10)

min=0.32, mean=0.32, max=0.32, sum=0.64 (2)

min=0.234, mean=0.234, max=0.234, sum=0.468 (2)

min=0.131, mean=0.131, max=0.131, sum=0.261 (2)

min=0.152, mean=0.193, max=0.238, sum=1.93 (10)

min=0.027, mean=0.039, max=0.054, sum=0.232 (6)

min=0.04, mean=0.063, max=0.101, sum=0.379 (6)

min=0.095, mean=0.111, max=0.128, sum=0.664 (6)

min=0.115, mean=0.137, max=0.173, sum=4.107 (30)

min=0.248, mean=0.248, max=0.248, sum=0.497 (2)

min=0.41, mean=0.41, max=0.41, sum=0.821 (2)

min=0.22, mean=0.336, max=0.41, sum=3.357 (10)

min=0.293, mean=0.293, max=0.293, sum=0.585 (2)

min=0.179, mean=0.179, max=0.179, sum=0.359 (2)

min=0.136, mean=0.136, max=0.136, sum=0.271 (2)

min=0.072, mean=0.146, max=0.215, sum=1.458 (10)

min=0.027, mean=0.041, max=0.049, sum=0.249 (6)

min=0.055, mean=0.058, max=0.062, sum=0.347 (6)

min=0.071, mean=0.085, max=0.103, sum=0.51 (6)

min=0.062, mean=0.115, max=0.149, sum=3.464 (30)

min=0.44, mean=0.44, max=0.44, sum=0.88 (2)

min=0.235, mean=0.235, max=0.235, sum=0.471 (2)

min=0.376, mean=0.376, max=0.376, sum=0.753 (2)

min=0.341, mean=0.435, max=0.497, sum=4.351 (10)

min=0.233, mean=0.233, max=0.233, sum=0.467 (2)

min=0.22, mean=0.22, max=0.22, sum=0.44 (2)

min=0.111, mean=0.111, max=0.111, sum=0.222 (2)

min=0.131, mean=0.198, max=0.256, sum=1.983 (10)

min=0.023, mean=0.031, max=0.035, sum=0.183 (6)

min=0.033, mean=0.044, max=0.05, sum=0.262 (6)

min=0.051, mean=0.058, max=0.068, sum=0.351 (6)

min=0.094, mean=0.122, max=0.145, sum=3.662 (30)

min=0.419, mean=0.419, max=0.419, sum=0.838 (2)

min=0.232, mean=0.232, max=0.232, sum=0.465 (2)

min=0.399, mean=0.399, max=0.399, sum=0.798 (2)

min=0.3, mean=0.382, max=0.506, sum=3.822 (10)

min=0.277, mean=0.277, max=0.277, sum=0.555 (2)

min=0.21, mean=0.21, max=0.21, sum=0.421 (2)

min=0.123, mean=0.123, max=0.123, sum=0.246 (2)

min=0.101, mean=0.195, max=0.298, sum=1.948 (10)

min=0.033, mean=0.041, max=0.05, sum=0.244 (6)

min=0.052, mean=0.062, max=0.069, sum=0.372 (6)

min=0.054, mean=0.058, max=0.061, sum=0.349 (6)

min=0.115, mean=0.147, max=0.194, sum=4.414 (30)

min=0.39, mean=0.39, max=0.39, sum=0.779 (2)

min=0.209, mean=0.209, max=0.209, sum=0.418 (2)