Vary the instructions and labels for input/output.

ablation_prompts

How many models this model outperform on average (over columns).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

ROUGE-2: Average ROUGE score [(Lin, 2004)](https://aclanthology.org/W04-1013/) based on 2-gram overlap.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

accuracy

min=0.303, mean=0.627, max=0.939, sum=15.06 (24)

min=0.288, mean=0.633, max=0.951, sum=7.598 (12)

min=0.679, mean=0.684, max=0.687, sum=4.104 (6)

min=0.947, mean=0.951, max=0.956, sum=5.704 (6)

min=0.057, mean=0.556, max=0.988, sum=6.668 (12)

min=0.697, mean=0.702, max=0.707, sum=4.21 (6)

min=0.94, mean=0.947, max=0.956, sum=5.68 (6)

min=0.316, mean=0.668, max=0.963, sum=8.012 (12)

min=0.257, mean=0.269, max=0.283, sum=1.612 (6)

min=0.102, mean=0.119, max=0.142, sum=0.715 (6)

min=0.924, mean=0.934, max=0.948, sum=11.208 (12)

min=0.926, mean=0.934, max=0.941, sum=5.602 (6)

min=0.682, mean=0.686, max=0.693, sum=8.237 (12)

min=0.685, mean=0.691, max=0.696, sum=4.144 (6)

min=0.142, mean=0.154, max=0.17, sum=1.854 (12)

min=0.142, mean=0.153, max=0.167, sum=0.917 (6)

min=0.101, mean=0.112, max=0.121, sum=0.673 (6)

min=0.106, mean=0.118, max=0.135, sum=0.705 (6)

min=0.439, mean=0.63, max=0.793, sum=15.124 (24)

min=0.394, mean=0.63, max=0.829, sum=7.556 (12)

min=0.6, mean=0.606, max=0.612, sum=3.633 (6)

min=0.919, mean=0.941, max=0.953, sum=5.648 (6)

min=0.078, mean=0.571, max=0.976, sum=6.854 (12)

min=0.618, mean=0.632, max=0.642, sum=3.794 (6)

min=0.932, mean=0.943, max=0.956, sum=5.656 (6)

min=0.402, mean=0.624, max=0.848, sum=7.485 (12)

min=0.221, mean=0.227, max=0.236, sum=1.36 (6)

min=0.133, mean=0.144, max=0.16, sum=0.863 (6)

min=0.936, mean=0.945, max=0.95, sum=11.344 (12)

min=0.934, mean=0.943, max=0.951, sum=5.66 (6)

min=0.61, mean=0.621, max=0.628, sum=7.456 (12)

min=0.627, mean=0.637, max=0.65, sum=3.821 (6)

min=0.052, mean=0.08, max=0.118, sum=0.956 (12)

min=0.079, mean=0.107, max=0.135, sum=0.641 (6)

min=0.126, mean=0.143, max=0.161, sum=0.858 (6)

min=0.129, mean=0.147, max=0.167, sum=0.88 (6)

min=0, mean=0.184, max=0.98, sum=2.205 (12)

min=0, mean=0.352, max=0.985, sum=8.439 (24)

min=0.129, mean=0.148, max=0.173, sum=0.889 (6)

min=0.205, mean=0.247, max=0.299, sum=1.482 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.165, mean=0.175, max=0.189, sum=1.051 (6)

min=0.314, mean=0.432, max=0.624, sum=2.59 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.429, max=0.972, sum=5.148 (12)

min=0.138, mean=0.162, max=0.186, sum=0.972 (6)

min=0.114, mean=0.116, max=0.117, sum=0.694 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0, max=0, sum=0 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.164, mean=0.209, max=0.29, sum=1.252 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.181, mean=0.207, max=0.26, sum=2.488 (12)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.175, mean=0.189, max=0.201, sum=1.137 (6)

min=0.171, mean=0.19, max=0.203, sum=2.277 (12)

min=0.12, mean=0.122, max=0.123, sum=0.73 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.121, mean=0.122, max=0.122, sum=1.459 (12)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.11, mean=0.114, max=0.119, sum=0.686 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.112, mean=0.116, max=0.118, sum=0.695 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.015, mean=0.509, max=1, sum=12.226 (24)

min=0.01, mean=0.517, max=1, sum=6.202 (12)

min=0.503, mean=0.519, max=0.537, sum=3.114 (6)

min=0.938, mean=0.942, max=0.947, sum=5.654 (6)

min=0.001, mean=0.505, max=1, sum=6.058 (12)

min=0.483, mean=0.499, max=0.517, sum=2.997 (6)

min=0.925, mean=0.933, max=0.942, sum=5.6 (6)

min=0.331, mean=0.586, max=0.878, sum=7.027 (12)

min=0.181, mean=0.19, max=0.201, sum=1.137 (6)

min=0.128, mean=0.145, max=0.166, sum=0.869 (6)

min=0.932, mean=0.939, max=0.946, sum=11.264 (12)

min=0.929, mean=0.934, max=0.941, sum=5.606 (6)

min=0.548, mean=0.559, max=0.57, sum=6.707 (12)

min=0.536, mean=0.553, max=0.563, sum=3.319 (6)

min=0.127, mean=0.131, max=0.135, sum=1.573 (12)

min=0.121, mean=0.131, max=0.141, sum=0.785 (6)

min=0.104, mean=0.11, max=0.114, sum=0.66 (6)

min=0.116, mean=0.134, max=0.156, sum=0.802 (6)

min=0.007, mean=0.506, max=1, sum=12.135 (24)

min=0.006, mean=0.5, max=1, sum=5.997 (12)

min=0.571, mean=0.576, max=0.582, sum=3.457 (6)

min=0.916, mean=0.929, max=0.944, sum=5.574 (6)

min=0.001, mean=0.501, max=1, sum=6.015 (12)

min=0.57, mean=0.576, max=0.581, sum=3.455 (6)

min=0.932, mean=0.937, max=0.942, sum=5.62 (6)

min=0.001, mean=0.501, max=1, sum=6.017 (12)

min=0.204, mean=0.211, max=0.225, sum=1.267 (6)

min=0.117, mean=0.13, max=0.152, sum=0.779 (6)

min=0.946, mean=0.948, max=0.95, sum=11.376 (12)

min=0.943, mean=0.945, max=0.949, sum=5.672 (6)

min=0.581, mean=0.596, max=0.608, sum=7.153 (12)

min=0.585, mean=0.597, max=0.607, sum=3.581 (6)

min=0.108, mean=0.123, max=0.138, sum=1.475 (12)

min=0.12, mean=0.129, max=0.136, sum=0.772 (6)

min=0.1, mean=0.109, max=0.125, sum=0.654 (6)

min=0.096, mean=0.101, max=0.108, sum=0.608 (6)

min=0.098, mean=0.503, max=0.915, sum=6.036 (12)

min=0.122, mean=0.512, max=0.902, sum=12.299 (24)

min=0.257, mean=0.442, max=0.55, sum=2.651 (6)

min=0.177, mean=0.317, max=0.522, sum=1.9 (6)

min=0.102, mean=0.483, max=0.854, sum=5.798 (12)

min=0.184, mean=0.427, max=0.562, sum=2.559 (6)

min=0.189, mean=0.325, max=0.538, sum=1.95 (6)

min=0.161, mean=0.505, max=0.866, sum=6.055 (12)

min=0.106, mean=0.134, max=0.152, sum=0.806 (6)

min=0.048, mean=0.048, max=0.048, sum=0.286 (6)

min=0.009, mean=0.015, max=0.02, sum=0.09 (6)

min=0.027, mean=0.035, max=0.049, sum=0.208 (6)

min=0.248, mean=0.379, max=0.568, sum=4.548 (12)

min=0.356, mean=0.466, max=0.53, sum=2.799 (6)

min=0.278, mean=0.477, max=0.588, sum=5.728 (12)

min=0.036, mean=0.036, max=0.036, sum=0.216 (6)

min=0.043, mean=0.043, max=0.043, sum=0.514 (12)

min=0.057, mean=0.057, max=0.057, sum=0.341 (6)

min=0.053, mean=0.053, max=0.053, sum=0.319 (6)

min=0.064, mean=0.526, max=0.951, sum=6.315 (12)

min=0.063, mean=0.523, max=0.976, sum=12.541 (24)

min=0.162, mean=0.195, max=0.216, sum=1.172 (6)

min=0.162, mean=0.338, max=0.522, sum=2.026 (6)

min=0.048, mean=0.506, max=0.951, sum=6.07 (12)

min=0.18, mean=0.325, max=0.404, sum=1.948 (6)

min=0.122, mean=0.343, max=0.562, sum=2.056 (6)

min=0.11, mean=0.505, max=0.871, sum=6.058 (12)

min=0.143, mean=0.176, max=0.195, sum=1.058 (6)

min=0.035, mean=0.035, max=0.035, sum=0.212 (6)

min=0.104, mean=0.162, max=0.265, sum=0.974 (6)

min=0.118, mean=0.293, max=0.508, sum=1.76 (6)

min=0.13, mean=0.337, max=0.556, sum=4.04 (12)

min=0.203, mean=0.366, max=0.449, sum=2.196 (6)

min=0.195, mean=0.349, max=0.432, sum=4.191 (12)

min=0.026, mean=0.026, max=0.026, sum=0.157 (6)

min=0.03, mean=0.03, max=0.03, sum=0.364 (12)

min=0.038, mean=0.038, max=0.039, sum=0.228 (6)

min=0.046, mean=0.047, max=0.047, sum=0.282 (6)

min=0, mean=0.501, max=1, sum=12.035 (24)

min=0.586, mean=0.594, max=0.604, sum=3.566 (6)

min=0.925, mean=0.929, max=0.936, sum=5.576 (6)

min=0.015, mean=0.509, max=1, sum=6.109 (12)

min=0.604, mean=0.608, max=0.612, sum=3.65 (6)

min=0.944, mean=0.953, max=0.965, sum=5.716 (6)

min=0.003, mean=0.506, max=1, sum=6.067 (12)

min=0.211, mean=0.219, max=0.226, sum=1.311 (6)

min=0.128, mean=0.142, max=0.156, sum=0.852 (6)

min=0.932, mean=0.947, max=0.96, sum=11.368 (12)

min=0.94, mean=0.948, max=0.956, sum=5.688 (6)

min=0.607, mean=0.615, max=0.619, sum=7.379 (12)

min=0.618, mean=0.621, max=0.624, sum=3.723 (6)

Accuracy

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

calibration

min=0.113, mean=0.209, max=0.449, sum=5.025 (24)

min=0.072, mean=0.214, max=0.445, sum=2.573 (12)

min=0.05, mean=0.074, max=0.096, sum=0.445 (6)

min=0.013, mean=0.032, max=0.06, sum=0.193 (6)

min=0.147, mean=0.353, max=0.621, sum=4.233 (12)

min=0.064, mean=0.083, max=0.108, sum=0.501 (6)

min=0.012, mean=0.021, max=0.025, sum=0.124 (6)

min=0.1, mean=0.213, max=0.437, sum=2.559 (12)

min=0.477, mean=0.497, max=0.518, sum=2.983 (6)

min=0.89, mean=0.925, max=0.944, sum=5.55 (6)

min=0.628, mean=0.726, max=0.812, sum=8.706 (12)

min=0.01, mean=0.015, max=0.024, sum=0.18 (12)

min=0.012, mean=0.024, max=0.05, sum=0.147 (6)

min=0.048, mean=0.067, max=0.088, sum=0.8 (12)

min=0.053, mean=0.071, max=0.09, sum=0.423 (6)

min=0.073, mean=0.28, max=0.392, sum=6.725 (24)

min=0.036, mean=0.291, max=0.468, sum=3.493 (12)

min=0.349, mean=0.36, max=0.382, sum=2.162 (6)

min=0.908, mean=0.929, max=0.943, sum=5.574 (6)

min=0.077, mean=0.568, max=0.975, sum=6.817 (12)

min=0.369, mean=0.373, max=0.378, sum=2.239 (6)

min=0.787, mean=0.818, max=0.847, sum=4.905 (6)

min=0.378, mean=0.593, max=0.81, sum=7.121 (12)

min=0.111, mean=0.116, max=0.121, sum=0.698 (6)

min=0.819, mean=0.824, max=0.828, sum=4.944 (6)

min=0.534, mean=0.552, max=0.576, sum=6.629 (12)

min=0.305, mean=0.343, max=0.41, sum=4.116 (12)

min=0.552, mean=0.646, max=0.713, sum=3.879 (6)

min=0.338, mean=0.347, max=0.36, sum=4.164 (12)

min=0.389, mean=0.397, max=0.401, sum=2.382 (6)

min=0.449, mean=0.513, max=0.581, sum=6.151 (12)

min=0.239, mean=0.421, max=0.574, sum=10.114 (24)

min=0.0, mean=0.0, max=0.0, sum=0.001 (6)

min=0.323, mean=0.352, max=0.382, sum=2.113 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.151, mean=0.494, max=0.84, sum=5.934 (12)

min=0.046, mean=0.147, max=0.225, sum=0.885 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.144, mean=0.429, max=0.765, sum=5.142 (12)

min=0.315, mean=0.333, max=0.344, sum=2.001 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.682, mean=0.756, max=0.81, sum=9.07 (12)

min=0.173, mean=0.283, max=0.361, sum=1.696 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.207, mean=0.291, max=0.36, sum=3.487 (12)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.0, mean=0.0, max=0.0, sum=0.001 (12)

min=0.337, mean=0.477, max=0.626, sum=11.446 (24)

min=0.203, mean=0.461, max=0.653, sum=5.536 (12)

min=0.321, mean=0.33, max=0.34, sum=1.977 (6)

min=0.232, mean=0.272, max=0.315, sum=1.632 (6)

min=0.365, mean=0.482, max=0.576, sum=5.781 (12)