Targeted evaluation of linguistic capabilities.

language

How many models this model outperform on average (over columns).

The Pile corpus for measuring lanugage model performance across various domains [(Gao et al., 2020)](https://arxiv.org/pdf/2101.00027.pdf).

Bits/byte: Average number of bits per byte according to model probabilities.

The TwitterAAE corpus of [Blodgett et al. (2016)](https://aclanthology.org/D16-1120/) for measuring language model performance in tweets as a function of speaker dialect.

Bits/byte: Average number of bits per byte according to model probabilities.

The International Corpus of English (ICE) drawn from English speakers from various places in the world, initiated by [Greenbaum (1991)](https://www.cambridge.org/core/journals/english-today/article/abs/ice-the-international-corpus-of-english/47808205394C538393C3FD8E62E5E701).

Bits/byte: Average number of bits per byte according to model probabilities.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

accuracy

min=0.28, mean=0.545, max=0.745, sum=3.268 (6)

min=1.829, mean=2.18, max=2.53, sum=4.359 (2)

min=0.693, mean=0.849, max=0.925, sum=5.096 (6)

min=0.738, mean=0.811, max=0.927, sum=3.244 (4)

min=0.358, mean=0.645, max=0.833, sum=3.87 (6)

min=1.888, mean=2.245, max=2.603, sum=4.491 (2)

min=0.739, mean=0.895, max=0.972, sum=5.368 (6)

min=0.75, mean=0.806, max=0.903, sum=3.224 (4)

min=0.314, mean=0.6, max=0.792, sum=3.603 (6)

min=1.852, mean=2.205, max=2.559, sum=4.411 (2)

min=0.714, mean=0.868, max=0.943, sum=5.21 (6)

min=0.727, mean=0.803, max=0.911, sum=3.213 (4)

min=0.333, mean=0.602, max=0.767, sum=3.611 (6)

min=1.824, mean=2.156, max=2.489, sum=4.312 (2)

min=0.685, mean=0.836, max=0.909, sum=5.019 (6)

min=0.728, mean=0.804, max=0.927, sum=3.218 (4)

min=0.241, mean=0.533, max=0.715, sum=3.198 (6)

min=1.798, mean=2.121, max=2.443, sum=4.242 (2)

min=0.661, mean=0.799, max=0.865, sum=4.795 (6)

min=0.736, mean=0.81, max=0.929, sum=3.239 (4)

min=0.322, mean=0.593, max=0.762, sum=3.555 (6)

min=1.813, mean=2.137, max=2.462, sum=4.275 (2)

min=0.682, mean=0.83, max=0.901, sum=4.977 (6)

min=0.733, mean=0.809, max=0.935, sum=3.235 (4)

min=0.343, mean=0.639, max=0.81, sum=3.832 (6)

min=1.841, mean=2.179, max=2.518, sum=4.359 (2)

min=0.715, mean=0.865, max=0.94, sum=5.192 (6)

min=0.737, mean=0.816, max=0.927, sum=3.264 (4)

min=0.336, mean=0.597, max=0.874, sum=3.583 (6)
☠ Anthropic is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2112.00861.

min=1.86, mean=2.109, max=2.359, sum=4.219 (2)

min=0.723, mean=0.822, max=0.88, sum=4.93 (6)

min=0.772, mean=0.829, max=0.937, sum=3.317 (4)

min=0.269, mean=0.571, max=0.828, sum=3.423 (6)
☠ BLOOM is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://huggingface.co/spaces/bigscience/BigScienceCorpus.

min=1.644, mean=1.986, max=2.328, sum=3.972 (2)

min=0.645, mean=0.715, max=0.765, sum=4.292 (6)

min=0.743, mean=0.819, max=0.929, sum=3.276 (4)

min=0.565, mean=0.757, max=0.936, sum=2.271 (3)

min=1.941, mean=2.305, max=2.669, sum=4.609 (2)

min=0.758, mean=0.875, max=0.951, sum=5.249 (6)

min=0.758, mean=0.83, max=0.953, sum=3.32 (4)

min=0.64, mean=0.811, max=0.984, sum=2.432 (3)

min=1.954, mean=2.324, max=2.693, sum=4.647 (2)

min=0.8, mean=0.916, max=0.993, sum=5.499 (6)

min=0.73, mean=0.833, max=0.962, sum=3.334 (4)

min=0.68, mean=0.845, max=1.017, sum=2.534 (3)

min=1.968, mean=2.341, max=2.715, sum=4.683 (2)

min=0.835, mean=0.954, max=1.029, sum=5.724 (6)

min=0.714, mean=0.823, max=0.953, sum=3.292 (4)

min=0.837, mean=0.996, max=1.182, sum=2.987 (3)

min=2.136, mean=2.525, max=2.913, sum=5.049 (2)

min=0.953, mean=1.112, max=1.205, sum=6.673 (6)

min=0.683, mean=0.793, max=0.927, sum=3.172 (4)

min=0.544, mean=0.741, max=0.932, sum=2.224 (3)

min=1.955, mean=2.325, max=2.694, sum=4.649 (2)

min=0.756, mean=0.923, max=1.016, sum=5.537 (6)

min=0.77, mean=0.832, max=0.934, sum=3.327 (4)

min=0.711, mean=0.858, max=1.049, sum=2.574 (3)

min=2.039, mean=2.428, max=2.817, sum=4.856 (2)

min=0.815, mean=1.009, max=1.114, sum=6.055 (6)

min=0.728, mean=0.832, max=0.961, sum=3.326 (4)

min=0.735, mean=0.875, max=1.053, sum=2.625 (3)

min=2.201, mean=2.61, max=3.019, sum=5.22 (2)

min=0.828, mean=1.024, max=1.159, sum=6.141 (6)

min=0.713, mean=0.79, max=0.904, sum=3.159 (4)

min=0.611, mean=0.781, max=0.961, sum=2.344 (3)

min=2.308, mean=2.73, max=3.151, sum=5.46 (2)

min=0.768, mean=0.943, max=1.06, sum=5.656 (6)

min=0.76, mean=0.805, max=0.91, sum=3.221 (4)

min=0.234, mean=0.49, max=0.701, sum=2.942 (6)
☠ GPT-J is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/.

min=1.675, mean=1.989, max=2.302, sum=3.977 (2)

min=0.68, mean=0.752, max=0.803, sum=4.513 (6)

min=0.725, mean=0.834, max=0.938, sum=3.336 (4)

min=0.212, mean=0.466, max=0.677, sum=2.796 (6)
☠ GPT-NeoX is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2204.06745.

min=1.632, mean=1.943, max=2.254, sum=3.886 (2)

min=0.659, mean=0.742, max=0.796, sum=4.453 (6)

min=0.742, mean=0.839, max=0.951, sum=3.354 (4)

min=0.443, mean=0.592, max=0.73, sum=3.554 (6)
☠ OPT is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2205.01068.

min=1.506, mean=1.81, max=2.114, sum=3.62 (2)

min=0.661, mean=0.741, max=0.794, sum=4.444 (6)

min=0.747, mean=0.831, max=0.892, sum=3.322 (4)

min=0.468, mean=0.618, max=0.756, sum=3.706 (6)
☠ OPT is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2205.01068.

min=1.57, mean=1.85, max=2.13, sum=3.7 (2)

min=0.677, mean=0.764, max=0.819, sum=4.581 (6)

min=0.741, mean=0.827, max=0.882, sum=3.307 (4)

min=0.319, mean=0.61, max=0.872, sum=3.661 (6)
☠ MT-NLG is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2201.11990.

min=1.903, mean=2.245, max=2.587, sum=4.49 (2)

min=0.753, mean=0.85, max=0.892, sum=5.1 (6)

min=0.779, mean=0.826, max=0.859, sum=3.302 (4)

min=0.418, mean=0.704, max=0.981, sum=4.226 (6)
☠ MT-NLG is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2201.11990.

min=1.972, mean=2.323, max=2.673, sum=4.645 (2)

min=0.818, mean=0.918, max=0.973, sum=5.51 (6)

min=0.731, mean=0.835, max=0.96, sum=3.339 (4)

min=0.515, mean=0.713, max=0.979, sum=4.278 (6)

min=1.806, mean=2.141, max=2.476, sum=4.282 (2)

min=0.842, mean=0.977, max=1.018, sum=5.861 (6)

min=0.721, mean=0.84, max=0.961, sum=3.359 (4)

min=0.587, mean=0.789, max=1.054, sum=4.735 (6)

min=1.864, mean=2.21, max=2.556, sum=4.42 (2)

min=0.83, mean=0.958, max=1.005, sum=5.748 (6)

min=0.725, mean=0.832, max=0.943, sum=3.326 (4)

min=0.687, mean=0.866, max=1.123, sum=5.194 (6)

min=1.936, mean=2.291, max=2.646, sum=4.582 (2)

min=0.885, mean=1.048, max=1.122, sum=6.29 (6)

min=0.68, mean=0.833, max=0.979, sum=3.331 (4)

min=0.823, mean=0.96, max=1.212, sum=5.763 (6)

min=2.025, mean=2.395, max=2.764, sum=4.789 (2)

min=0.948, mean=1.136, max=1.221, sum=6.818 (6)

min=0.692, mean=0.818, max=0.973, sum=3.271 (4)

min=0.205, mean=0.576, max=0.863, sum=3.455 (6)

min=1.862, mean=2.192, max=2.522, sum=4.384 (2)

min=0.72, mean=0.838, max=0.895, sum=5.028 (6)

min=0.755, mean=0.823, max=0.951, sum=3.29 (4)

min=0.214, mean=0.578, max=0.827, sum=3.466 (6)

min=1.815, mean=2.147, max=2.478, sum=4.293 (2)

min=0.726, mean=0.823, max=0.874, sum=4.937 (6)

min=0.74, mean=0.812, max=0.89, sum=3.248 (4)

min=0.677, mean=0.969, max=1.275, sum=5.811 (6)

min=2.221, mean=2.628, max=3.034, sum=5.255 (2)

min=1.022, mean=1.192, max=1.269, sum=7.153 (6)

min=0.68, mean=0.761, max=0.812, sum=3.042 (4)

min=0.814, mean=1.103, max=1.435, sum=6.62 (6)

min=2.506, mean=2.946, max=3.385, sum=5.892 (2)

min=1.13, mean=1.326, max=1.401, sum=7.957 (6)

min=0.698, mean=0.769, max=0.823, sum=3.078 (4)

min=1.156, mean=1.61, max=2.003, sum=9.661 (6)

min=3.048, mean=3.505, max=3.963, sum=7.011 (2)

min=1.672, mean=1.975, max=2.065, sum=11.85 (6)

min=0.648, mean=0.728, max=0.774, sum=2.911 (4)

Accuracy

The Pile corpus for measuring lanugage model performance across various domains [(Gao et al., 2020)](https://arxiv.org/pdf/2101.00027.pdf).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The TwitterAAE corpus of [Blodgett et al. (2016)](https://aclanthology.org/D16-1120/) for measuring language model performance in tweets as a function of speaker dialect.

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The International Corpus of English (ICE) drawn from English speakers from various places in the world, initiated by [Greenbaum (1991)](https://www.cambridge.org/core/journals/english-today/article/abs/ice-the-international-corpus-of-english/47808205394C538393C3FD8E62E5E701).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

efficiency

min=0.422, mean=0.848, max=1.13, sum=5.085 (6)

min=0.263, mean=0.263, max=0.263, sum=0.525 (2)

min=0.984, mean=1.104, max=1.13, sum=6.625 (6)

min=0.259, mean=0.26, max=0.26, sum=1.04 (4)

min=0.35, mean=0.642, max=0.835, sum=3.852 (6)

min=0.24, mean=0.24, max=0.24, sum=0.481 (2)

min=0.736, mean=0.818, max=0.835, sum=4.907 (6)

min=0.238, mean=0.239, max=0.239, sum=0.955 (4)

min=0.426, mean=0.753, max=0.969, sum=4.52 (6)

min=0.287, mean=0.288, max=0.288, sum=0.575 (2)

min=0.852, mean=0.948, max=0.968, sum=5.688 (6)

min=0.281, mean=0.283, max=0.283, sum=1.131 (4)

min=0.771, mean=2.911, max=9.758, sum=17.464 (6)
☠ Anthropic is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2112.00861.

min=0.731, mean=0.736, max=0.742, sum=1.473 (2)

min=1.423, mean=1.848, max=3.327, sum=11.09 (6)

min=0.447, mean=0.452, max=0.455, sum=1.808 (4)

min=0.628, mean=0.969, max=1.172, sum=5.813 (6)
☠ BLOOM is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://huggingface.co/spaces/bigscience/BigScienceCorpus.

min=0.035, mean=0.036, max=0.036, sum=0.071 (2)

min=1.141, mean=1.146, max=1.156, sum=6.874 (6)

min=0.041, mean=0.448, max=1.093, sum=1.793 (4)

min=0.471, mean=0.654, max=0.944, sum=1.961 (3)

min=0.319, mean=0.32, max=0.321, sum=0.64 (2)

min=0.932, mean=0.944, max=0.947, sum=5.666 (6)

min=0.314, mean=0.316, max=0.316, sum=1.263 (4)

min=0.31, mean=0.456, max=0.69, sum=1.367 (3)

min=0.204, mean=0.205, max=0.205, sum=0.409 (2)

min=0.68, mean=0.69, max=0.692, sum=4.139 (6)

min=0.202, mean=0.202, max=0.203, sum=0.81 (4)

min=0.268, mean=0.37, max=0.533, sum=1.11 (3)

min=0.189, mean=0.189, max=0.19, sum=0.379 (2)

min=0.526, mean=0.533, max=0.534, sum=3.195 (6)

min=0.187, mean=0.188, max=0.188, sum=0.751 (4)

min=0.283, mean=0.394, max=0.572, sum=1.181 (3)

min=0.215, mean=0.215, max=0.215, sum=0.43 (2)

min=0.564, mean=0.571, max=0.573, sum=3.429 (6)

min=0.214, mean=0.214, max=0.214, sum=0.856 (4)

min=0.45, mean=0.455, max=0.458, sum=2.727 (6)
☠ GPT-J is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arankomatsuzaki.wordpress.com/2021/06/04/gpt-j/.

min=0.437, mean=0.438, max=0.438, sum=0.875 (2)

min=0.486, mean=0.511, max=0.553, sum=3.067 (6)

min=0.504, mean=0.541, max=0.59, sum=2.164 (4)

min=0.85, mean=0.902, max=0.94, sum=5.412 (6)
☠ GPT-NeoX is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2204.06745.

min=0.824, mean=0.824, max=0.824, sum=1.648 (2)

min=0.908, mean=0.951, max=1.006, sum=5.704 (6)

min=0.012, mean=0.48, max=1.021, sum=1.918 (4)

min=0.751, mean=0.829, max=0.884, sum=4.975 (6)
☠ OPT is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2205.01068.

min=0.134, mean=0.184, max=0.233, sum=0.368 (2)

min=0.932, mean=0.994, max=1.015, sum=5.964 (6)

min=0.048, mean=0.517, max=0.986, sum=2.068 (4)

min=0.323, mean=0.495, max=0.782, sum=2.973 (6)
☠ OPT is explicitly trained on the Pile, i.e. data from the same distribution as the test set. See https://arxiv.org/abs/2205.01068.

min=1.262, mean=1.274, max=1.285, sum=2.548 (2)

min=0.324, mean=0.331, max=0.339, sum=1.985 (6)

min=0.103, mean=0.194, max=0.318, sum=0.777 (4)

min=0.205, mean=0.226, max=0.239, sum=1.359 (6)

min=0.185, mean=0.185, max=0.185, sum=0.37 (2)

min=0.238, mean=0.239, max=0.239, sum=1.433 (6)

min=0.184, mean=0.184, max=0.184, sum=0.736 (4)

min=0.091, mean=0.109, max=0.12, sum=0.657 (6)

min=0.08, mean=0.08, max=0.08, sum=0.16 (2)

min=0.119, mean=0.12, max=0.12, sum=0.721 (6)