Vary the number of in-context training examples.

ablation_in_context

How many models this model outperform on average (over columns).

The NaturalQuestions [(Kwiatkowski et al., 2019)](https://aclanthology.org/Q19-1026/) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

F1: Average F1 score in terms of word overlap between the model output and correct reference.

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

ROUGE-2: Average ROUGE score [(Lin, 2004)](https://aclanthology.org/W04-1013/) based on 2-gram overlap.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

accuracy

min=0.429, mean=0.429, max=0.429, sum=0.858 (2)

min=0.13, mean=0.13, max=0.13, sum=0.259 (2)

min=0.595, mean=0.595, max=0.595, sum=1.19 (2)

min=0.633, mean=0.68, max=0.708, sum=4.08 (6)

min=0.003, mean=0.005, max=0.007, sum=0.027 (6)

min=0.92, mean=0.93, max=0.936, sum=5.58 (6)

min=0.168, mean=0.597, max=0.951, sum=7.159 (12)

min=0.691, mean=0.696, max=0.702, sum=4.178 (6)

min=0.143, mean=0.152, max=0.157, sum=0.912 (6)

min=0.945, mean=0.949, max=0.952, sum=5.694 (6)

min=0.695, mean=0.701, max=0.713, sum=4.208 (6)

min=0.155, mean=0.159, max=0.163, sum=0.955 (6)

min=0.929, mean=0.935, max=0.943, sum=5.612 (6)

min=0, mean=0.571, max=0.996, sum=6.856 (12)

min=0.681, mean=0.693, max=0.706, sum=4.159 (6)

min=0.157, mean=0.162, max=0.17, sum=0.97 (6)

min=0.933, mean=0.94, max=0.952, sum=5.64 (6)

min=0.012, mean=0.569, max=1, sum=6.829 (12)

min=0.682, mean=0.692, max=0.702, sum=4.154 (6)

min=0.148, mean=0.153, max=0.159, sum=0.916 (6)

min=0.939, mean=0.945, max=0.952, sum=5.672 (6)

min=0.012, mean=0.536, max=0.997, sum=6.432 (12)

min=0.456, mean=0.456, max=0.456, sum=0.912 (2)

min=0.038, mean=0.038, max=0.038, sum=0.076 (2)

min=0.579, mean=0.636, max=0.668, sum=3.819 (6)

min=0.004, mean=0.005, max=0.007, sum=0.032 (6)

min=0.862, mean=0.893, max=0.928, sum=5.356 (6)

min=0, mean=0.498, max=0.997, sum=5.98 (12)

min=0.626, mean=0.628, max=0.631, sum=3.768 (6)

min=0.044, mean=0.074, max=0.106, sum=0.444 (6)

min=0.931, mean=0.943, max=0.95, sum=5.656 (6)

min=0.268, mean=0.61, max=0.886, sum=7.319 (12)

min=0.609, mean=0.641, max=0.66, sum=3.847 (6)

min=0.084, mean=0.1, max=0.128, sum=0.6 (6)

min=0.873, mean=0.913, max=0.935, sum=5.476 (6)

min=0.366, mean=0.611, max=0.813, sum=7.334 (12)

min=0.611, mean=0.623, max=0.632, sum=3.736 (6)

min=0.05, mean=0.091, max=0.153, sum=0.544 (6)

min=0.932, mean=0.942, max=0.949, sum=5.65 (6)

min=0.146, mean=0.605, max=0.956, sum=7.262 (12)

min=0.931, mean=0.944, max=0.951, sum=5.664 (6)

min=0.329, mean=0.613, max=0.884, sum=7.356 (12)

min=0.293, mean=0.293, max=0.293, sum=0.587 (2)

min=0.128, mean=0.128, max=0.128, sum=0.256 (2)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0, max=0, sum=0 (2)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.225, mean=0.239, max=0.264, sum=1.432 (6)

min=0.13, mean=0.131, max=0.131, sum=0.784 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.001, max=0.002, sum=0.006 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.002, max=0.009, sum=0.023 (12)

min=0.171, mean=0.19, max=0.203, sum=1.138 (6)

min=0.121, mean=0.122, max=0.122, sum=0.73 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.181, mean=0.207, max=0.26, sum=1.244 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.229, max=0.669, sum=2.744 (12)

min=0.193, mean=0.207, max=0.234, sum=1.24 (6)

min=0.11, mean=0.118, max=0.124, sum=0.708 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.002, mean=0.077, max=0.162, sum=0.46 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.255, max=0.862, sum=3.06 (12)

min=0.173, mean=0.19, max=0.203, sum=1.139 (6)

min=0.181, mean=0.208, max=0.262, sum=1.248 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.318, max=0.966, sum=3.813 (12)

min=0, mean=0.293, max=0.764, sum=3.514 (12)

min=0.412, mean=0.412, max=0.412, sum=0.823 (2)

min=0.105, mean=0.105, max=0.105, sum=0.211 (2)

min=0.511, mean=0.532, max=0.567, sum=3.192 (6)

min=0.046, mean=0.05, max=0.055, sum=0.299 (6)

min=0.923, mean=0.929, max=0.935, sum=5.574 (6)

min=0.535, mean=0.551, max=0.559, sum=3.303 (6)

min=0.116, mean=0.133, max=0.144, sum=0.797 (6)

min=0.938, mean=0.941, max=0.944, sum=5.646 (6)

min=0.161, mean=0.549, max=0.902, sum=6.582 (12)

min=0.541, mean=0.553, max=0.563, sum=3.32 (6)

min=0.135, mean=0.136, max=0.137, sum=0.817 (6)

min=0.934, mean=0.937, max=0.938, sum=5.62 (6)

min=0.024, mean=0.53, max=0.977, sum=6.357 (12)

min=0.533, mean=0.555, max=0.568, sum=3.328 (6)

min=0.121, mean=0.138, max=0.158, sum=0.825 (6)

min=0.937, mean=0.942, max=0.949, sum=5.652 (6)

min=0.167, mean=0.548, max=0.951, sum=6.574 (12)

min=0.938, mean=0.94, max=0.944, sum=5.64 (6)

min=0.199, mean=0.555, max=0.915, sum=6.655 (12)

min=0.446, mean=0.446, max=0.446, sum=0.893 (2)

min=0.096, mean=0.096, max=0.096, sum=0.192 (2)

min=0.561, mean=0.586, max=0.607, sum=3.513 (6)

min=0.003, mean=0.003, max=0.004, sum=0.02 (6)

min=0.918, mean=0.933, max=0.943, sum=5.598 (6)

min=0.598, mean=0.606, max=0.61, sum=3.634 (6)

min=0.125, mean=0.136, max=0.143, sum=0.818 (6)

min=0.944, mean=0.948, max=0.952, sum=5.686 (6)

min=0.146, mean=0.545, max=0.939, sum=6.544 (12)

min=0.587, mean=0.595, max=0.611, sum=3.572 (6)

min=0.14, mean=0.144, max=0.148, sum=0.862 (6)

min=0.947, mean=0.951, max=0.955, sum=5.704 (6)

min=0.012, mean=0.566, max=0.996, sum=6.792 (12)

min=0.566, mean=0.592, max=0.607, sum=3.553 (6)

min=0.121, mean=0.136, max=0.149, sum=0.814 (6)

min=0.944, mean=0.949, max=0.953, sum=5.692 (6)

min=0.095, mean=0.535, max=0.939, sum=6.421 (12)

min=0.947, mean=0.949, max=0.952, sum=5.692 (6)

min=0, mean=0.557, max=0.982, sum=6.685 (12)

min=0.247, mean=0.247, max=0.247, sum=0.494 (2)

min=0.044, mean=0.044, max=0.044, sum=0.087 (2)

min=0.278, mean=0.486, max=0.593, sum=2.918 (6)

min=0.043, mean=0.043, max=0.043, sum=0.257 (6)

min=0.248, mean=0.38, max=0.57, sum=2.278 (6)

min=0, mean=0.499, max=0.997, sum=5.988 (12)

min=0.278, mean=0.477, max=0.588, sum=2.864 (6)

min=0.248, mean=0.379, max=0.568, sum=2.274 (6)

min=0.171, mean=0.517, max=0.857, sum=6.198 (12)

min=0, mean=0.497, max=0.999, sum=5.969 (12)

min=0.012, mean=0.506, max=0.993, sum=6.069 (12)

min=0.166, mean=0.166, max=0.166, sum=0.333 (2)

min=0.031, mean=0.031, max=0.031, sum=0.061 (2)

min=0.195, mean=0.352, max=0.432, sum=2.114 (6)

min=0.03, mean=0.03, max=0.03, sum=0.182 (6)

min=0.13, mean=0.336, max=0.554, sum=2.016 (6)

min=0, mean=0.499, max=0.997, sum=5.984 (12)

min=0.195, mean=0.349, max=0.432, sum=2.095 (6)

min=0.13, mean=0.337, max=0.556, sum=2.02 (6)

min=0.044, mean=0.511, max=0.976, sum=6.137 (12)

min=0.079, mean=0.518, max=0.976, sum=6.22 (12)

min=0.067, mean=0.512, max=0.976, sum=6.139 (12)

min=0.458, mean=0.458, max=0.458, sum=0.916 (2)

min=0.117, mean=0.117, max=0.117, sum=0.233 (2)

min=0.001, mean=0.001, max=0.001, sum=0.002 (2)

min=0.577, mean=0.597, max=0.612, sum=3.579 (6)

min=0.001, mean=0.001, max=0.001, sum=0.008 (6)

min=0.892, mean=0.908, max=0.917, sum=5.45 (6)

min=0, mean=0.499, max=0.999, sum=5.983 (12)

min=0.613, mean=0.616, max=0.618, sum=3.695 (6)

min=0.15, mean=0.154, max=0.159, sum=0.923 (6)

min=0.934, mean=0.947, max=0.96, sum=5.684 (6)

min=0.007, mean=0.526, max=1, sum=6.311 (12)

min=0.604, mean=0.611, max=0.616, sum=3.666 (6)

min=0.153, mean=0.158, max=0.161, sum=0.947 (6)

min=0.935, mean=0.939, max=0.943, sum=5.634 (6)

min=0.003, mean=0.505, max=1, sum=6.059 (12)

min=0.602, mean=0.617, max=0.626, sum=3.704 (6)

min=0.147, mean=0.158, max=0.168, sum=0.945 (6)

min=0.935, mean=0.946, max=0.958, sum=5.674 (6)

min=0.001, mean=0.505, max=1, sum=6.055 (12)

min=0.934, mean=0.947, max=0.959, sum=5.682 (6)

min=0.009, mean=0.514, max=1, sum=6.169 (12)

min=0.438, mean=0.438, max=0.438, sum=0.876 (2)

min=0.552, mean=0.571, max=0.598, sum=3.424 (6)

min=0.002, mean=0.003, max=0.003, sum=0.015 (6)

min=0.857, mean=0.871, max=0.882, sum=5.226 (6)

min=0, mean=0.507, max=0.988, sum=6.079 (12)

min=0.589, mean=0.602, max=0.615, sum=3.612 (6)

min=0.139, mean=0.143, max=0.152, sum=0.861 (6)

min=0.92, mean=0.926, max=0.931, sum=5.554 (6)

min=0.038, mean=0.515, max=0.963, sum=6.175 (12)

min=0.573, mean=0.584, max=0.604, sum=3.506 (6)

min=0.144, mean=0.146, max=0.147, sum=0.876 (6)

min=0.911, mean=0.914, max=0.916, sum=5.484 (6)

min=0.001, mean=0.504, max=1, sum=6.047 (12)

min=0.565, mean=0.598, max=0.624, sum=3.587 (6)

min=0.138, mean=0.144, max=0.154, sum=0.862 (6)

min=0.915, mean=0.924, max=0.932, sum=5.544 (6)

min=0.92, mean=0.928, max=0.937, sum=5.566 (6)

min=0.069, mean=0.525, max=0.976, sum=6.302 (12)

min=0.163, mean=0.163, max=0.163, sum=0.327 (2)

min=0.021, mean=0.021, max=0.021, sum=0.042 (2)

min=0.649, mean=0.663, max=0.672, sum=3.978 (6)

min=0.004, mean=0.005, max=0.006, sum=0.028 (6)

min=0.953, mean=0.963, max=0.97, sum=5.776 (6)

min=0.637, mean=0.642, max=0.65, sum=3.853 (6)

min=0.144, mean=0.155, max=0.164, sum=0.931 (6)

min=0.959, mean=0.96, max=0.962, sum=5.762 (6)

min=0.646, mean=0.651, max=0.654, sum=3.905 (6)

min=0.13, mean=0.139, max=0.146, sum=0.836 (6)

min=0.954, mean=0.958, max=0.962, sum=5.746 (6)

min=0.646, mean=0.655, max=0.662, sum=3.928 (6)

min=0.108, mean=0.14, max=0.159, sum=0.839 (6)

min=0.955, mean=0.956, max=0.957, sum=5.736 (6)

min=0.959, mean=0.961, max=0.963, sum=5.768 (6)

min=0.377, mean=0.377, max=0.377, sum=0.754 (2)

min=0.007, mean=0.007, max=0.007, sum=0.015 (2)

min=0.01, mean=0.01, max=0.01, sum=0.02 (2)

min=0.186, mean=0.216, max=0.247, sum=1.295 (6)

min=0.007, mean=0.007, max=0.008, sum=0.044 (6)

min=0.758, mean=0.796, max=0.821, sum=4.774 (6)

min=0, mean=0.469, max=0.999, sum=5.622 (12)

min=0.22, mean=0.237, max=0.258, sum=1.422 (6)

min=0.02, mean=0.022, max=0.023, sum=0.13 (6)

min=0.802, mean=0.834, max=0.857, sum=5.006 (6)

min=0.428, mean=0.54, max=0.63, sum=6.485 (12)

min=0.224, mean=0.24, max=0.267, sum=1.443 (6)

min=0.019, mean=0.019, max=0.02, sum=0.117 (6)

min=0.883, mean=0.894, max=0.903, sum=5.362 (6)

min=0.037, mean=0.492, max=0.952, sum=5.901 (12)

min=0.213, mean=0.22, max=0.233, sum=1.323 (6)

min=0.013, mean=0.02, max=0.026, sum=0.12 (6)

min=0.802, mean=0.855, max=0.882, sum=5.132 (6)