calibration

How many models this model outperform on average (over columns).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Max prob: Model's average confidence in its prediction (only computed for classification tasks)

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

1-bin expected calibration error: The (absolute value) difference between the model's average confidence and accuracy (only computed for classification tasks).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Selective coverage-accuracy area: The area under the coverage-accuracy curve, a standard selective classification metric (only computed for classification tasks).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Accuracy at 10% coverage: The accuracy for the 10% of predictions that the model is most confident on (only computed for classification tasks).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

1-bin expected calibration error (after Platt scaling): 1-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

10-bin Expected Calibration Error (after Platt scaling): 10-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Platt Scaling Coefficient: Coefficient of the Platt scaling classifier (can compare this across tasks).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Platt Scaling Intercept: Intercept of the Platt scaling classifier (can compare this across tasks).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Max prob: Model's average confidence in its prediction (only computed for classification tasks)

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

1-bin expected calibration error: The (absolute value) difference between the model's average confidence and accuracy (only computed for classification tasks).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Selective coverage-accuracy area: The area under the coverage-accuracy curve, a standard selective classification metric (only computed for classification tasks).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Accuracy at 10% coverage: The accuracy for the 10% of predictions that the model is most confident on (only computed for classification tasks).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

1-bin expected calibration error (after Platt scaling): 1-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

10-bin Expected Calibration Error (after Platt scaling): 10-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Platt Scaling Coefficient: Coefficient of the Platt scaling classifier (can compare this across tasks).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Platt Scaling Intercept: Intercept of the Platt scaling classifier (can compare this across tasks).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Max prob: Model's average confidence in its prediction (only computed for classification tasks)

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

1-bin expected calibration error: The (absolute value) difference between the model's average confidence and accuracy (only computed for classification tasks).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Selective coverage-accuracy area: The area under the coverage-accuracy curve, a standard selective classification metric (only computed for classification tasks).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Accuracy at 10% coverage: The accuracy for the 10% of predictions that the model is most confident on (only computed for classification tasks).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

1-bin expected calibration error (after Platt scaling): 1-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

10-bin Expected Calibration Error (after Platt scaling): 10-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Platt Scaling Coefficient: Coefficient of the Platt scaling classifier (can compare this across tasks).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Platt Scaling Intercept: Intercept of the Platt scaling classifier (can compare this across tasks).

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Max prob: Model's average confidence in its prediction (only computed for classification tasks)

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

1-bin expected calibration error: The (absolute value) difference between the model's average confidence and accuracy (only computed for classification tasks).

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Selective coverage-accuracy area: The area under the coverage-accuracy curve, a standard selective classification metric (only computed for classification tasks).

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Accuracy at 10% coverage: The accuracy for the 10% of predictions that the model is most confident on (only computed for classification tasks).

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

1-bin expected calibration error (after Platt scaling): 1-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

10-bin Expected Calibration Error (after Platt scaling): 10-bin ECE computed after applying Platt scaling to recalibrate the model's predicted probabilities.

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Platt Scaling Coefficient: Coefficient of the Platt scaling classifier (can compare this across tasks).

The CivilComments benchmark for toxicity detection [(Borkan et al., 2019)](https://arxiv.org/pdf/1903.04561.pdf).

Platt Scaling Intercept: Intercept of the Platt scaling classifier (can compare this across tasks).

calibration_detailed

min=0.302, mean=0.328, max=0.344, sum=4.921 (15)

min=0.004, mean=0.07, max=0.153, sum=1.051 (15)

min=0.074, mean=0.131, max=0.172, sum=1.96 (15)

min=0.185, mean=0.299, max=0.434, sum=4.478 (15)

min=0, mean=0.344, max=0.636, sum=5.167 (15)

min=0.057, mean=0.098, max=0.135, sum=1.467 (15)

min=-1.964, mean=0.869, max=2.587, sum=13.04 (15)

min=-2.707, mean=-0.523, max=1.087, sum=-7.846 (15)

min=0.874, mean=0.88, max=0.885, sum=2.639 (3)

min=0.06, mean=0.064, max=0.072, sum=0.191 (3)

min=0.983, mean=0.986, max=0.988, sum=2.959 (3)

min=0.012, mean=0.013, max=0.014, sum=0.039 (3)

min=1.324, mean=1.408, max=1.493, sum=4.223 (3)

min=0.142, mean=0.389, max=0.694, sum=1.168 (3)

min=0.405, mean=0.68, max=0.82, sum=22.433 (33)

min=0.002, mean=0.144, max=0.595, sum=4.741 (33)

min=0.103, mean=0.228, max=0.595, sum=7.528 (33)

min=0.157, mean=0.751, max=0.992, sum=24.769 (33)

min=0.043, mean=0.14, max=0.282, sum=4.632 (33)

min=-1.176, mean=0.834, max=3.436, sum=27.506 (33)

min=-1.619, mean=0.48, max=4.317, sum=15.841 (33)

min=0.529, mean=0.574, max=0.641, sum=30.996 (54)

min=0.001, mean=0.258, max=0.587, sum=13.95 (54)

min=0.048, mean=0.27, max=0.587, sum=14.569 (54)

min=0.003, mean=0.562, max=0.997, sum=30.344 (54)

min=0, mean=0.559, max=1, sum=30.197 (54)

min=0.004, mean=0.05, max=0.115, sum=2.676 (54)

min=-13.349, mean=0.939, max=18.049, sum=50.687 (54)

min=-1.815, mean=0.037, max=1.285, sum=1.995 (54)

min=0.294, mean=0.308, max=0.326, sum=4.619 (15)

min=0.004, mean=0.067, max=0.116, sum=1.01 (15)

min=0.051, mean=0.123, max=0.181, sum=1.842 (15)

min=0.18, mean=0.263, max=0.354, sum=3.939 (15)

min=0, mean=0.285, max=0.455, sum=4.273 (15)

min=0.052, mean=0.093, max=0.122, sum=1.391 (15)

min=-2.989, mean=0.663, max=4.74, sum=9.942 (15)

min=-3.632, mean=-0.595, max=2.573, sum=-8.929 (15)

min=0.717, mean=0.743, max=0.76, sum=2.23 (3)

min=0.196, mean=0.213, max=0.234, sum=0.639 (3)

min=0.991, mean=0.991, max=0.992, sum=2.973 (3)

min=0.007, mean=0.008, max=0.008, sum=0.023 (3)

min=2.107, mean=2.408, max=2.856, sum=7.223 (3)

min=1.201, mean=1.349, max=1.525, sum=4.046 (3)

min=0.403, mean=0.595, max=0.803, sum=19.637 (33)

min=0.002, mean=0.188, max=0.513, sum=6.196 (33)

min=0.134, mean=0.269, max=0.513, sum=8.875 (33)

min=0.187, mean=0.626, max=0.99, sum=20.655 (33)

min=0.2, mean=0.673, max=1, sum=22.2 (33)

min=0.022, mean=0.152, max=0.319, sum=5.011 (33)

min=-1.369, mean=1.632, max=12.809, sum=53.865 (33)

min=-7.129, mean=-0.587, max=3.683, sum=-19.367 (33)

min=0.502, mean=0.543, max=0.623, sum=29.314 (54)

min=0.013, mean=0.361, max=0.573, sum=19.521 (54)

min=0.073, mean=0.377, max=0.573, sum=20.347 (54)

min=0, mean=0.525, max=1.0, sum=28.362 (54)

min=0, mean=0.518, max=1, sum=27.946 (54)

min=0, mean=0.035, max=0.131, sum=1.903 (54)

min=-979.241, mean=-11.72, max=305.817, sum=-621.174 (53)

min=-121.276, mean=-1.66, max=26.334, sum=-87.986 (53)

min=0.288, mean=0.322, max=0.346, sum=4.837 (15)

min=0.002, mean=0.057, max=0.122, sum=0.86 (15)

min=0.063, mean=0.114, max=0.154, sum=1.708 (15)

min=0.168, mean=0.324, max=0.45, sum=4.853 (15)

min=0.091, mean=0.378, max=0.636, sum=5.674 (15)

min=0.052, mean=0.091, max=0.125, sum=1.37 (15)

min=-1.159, mean=2.683, max=7.151, sum=40.24 (15)

min=-2.297, mean=0.995, max=3.957, sum=14.926 (15)

min=0.791, mean=0.795, max=0.8, sum=2.386 (3)

min=0.152, mean=0.158, max=0.166, sum=0.473 (3)

min=0.986, mean=0.987, max=0.988, sum=2.961 (3)

min=0.99, mean=0.99, max=0.99, sum=2.97 (3)

min=0.005, mean=0.011, max=0.014, sum=0.032 (3)

min=2.019, mean=2.079, max=2.121, sum=6.238 (3)

min=0.725, mean=0.899, max=1.055, sum=2.697 (3)

min=0.502, mean=0.59, max=0.76, sum=19.467 (33)

min=0.027, mean=0.192, max=0.423, sum=6.331 (33)

min=0.113, mean=0.244, max=0.466, sum=8.048 (33)

min=0.078, mean=0.744, max=0.992, sum=24.542 (33)

min=0.033, mean=0.14, max=0.224, sum=4.607 (33)

min=-1.727, mean=1.621, max=7.736, sum=53.489 (33)

min=-2.554, mean=0.328, max=3.251, sum=10.83 (33)

min=0.543, mean=0.596, max=0.66, sum=32.192 (54)

min=0.209, mean=0.402, max=0.593, sum=21.7 (54)

min=0.228, mean=0.408, max=0.593, sum=22.008 (54)

min=0.001, mean=0.515, max=1.0, sum=27.793 (54)

min=0, mean=0.507, max=1, sum=27.402 (54)

min=0.001, mean=0.03, max=0.121, sum=1.601 (54)

min=-22.747, mean=0.092, max=14.162, sum=4.955 (54)

min=-3.072, mean=0.03, max=2.685, sum=1.606 (54)

min=0.338, mean=0.487, max=0.657, sum=7.308 (15)

min=0.007, mean=0.109, max=0.182, sum=1.636 (15)

min=0.067, mean=0.139, max=0.205, sum=2.09 (15)

min=0.292, mean=0.558, max=0.918, sum=8.365 (15)

min=0.364, mean=0.701, max=1, sum=10.515 (15)

min=0.056, mean=0.091, max=0.14, sum=1.363 (15)

min=0.094, mean=1.006, max=1.997, sum=15.095 (15)

min=-0.881, mean=-0.171, max=1.022, sum=-2.566 (15)

min=0.796, mean=0.822, max=0.841, sum=2.465 (3)

min=0.119, mean=0.136, max=0.165, sum=0.407 (3)

min=0.987, mean=0.992, max=0.996, sum=2.976 (3)

min=0.007, mean=0.009, max=0.012, sum=0.026 (3)

min=1.585, mean=1.949, max=2.167, sum=5.846 (3)

min=0.541, mean=0.78, max=0.954, sum=2.339 (3)

min=0.509, mean=0.628, max=0.788, sum=20.72 (33)

min=0.014, mean=0.169, max=0.473, sum=5.572 (33)

min=0.095, mean=0.234, max=0.473, sum=7.733 (33)

min=0.163, mean=0.772, max=0.997, sum=25.492 (33)

min=0.011, mean=0.129, max=0.275, sum=4.244 (33)

min=-1.923, mean=2.587, max=22.927, sum=85.37 (33)

min=-2.36, mean=0.05, max=2.173, sum=1.665 (33)

min=0.511, mean=0.6, max=0.691, sum=32.421 (54)

min=0.002, mean=0.368, max=0.649, sum=19.892 (54)

min=0.131, mean=0.376, max=0.649, sum=20.307 (54)

min=0.0, mean=0.538, max=1, sum=29.045 (54)

min=0, mean=0.034, max=0.145, sum=1.816 (54)

min=-119.594, mean=-1.658, max=19.993, sum=-87.869 (53)

min=-7.277, mean=0.041, max=2.96, sum=2.199 (53)

min=0.343, mean=0.558, max=0.78, sum=8.366 (15)

min=0.022, mean=0.096, max=0.248, sum=1.435 (15)

min=0.056, mean=0.137, max=0.248, sum=2.059 (15)

min=0.239, mean=0.597, max=0.936, sum=8.95 (15)

min=0.273, mean=0.727, max=1, sum=10.902 (15)

min=0.039, mean=0.088, max=0.15, sum=1.323 (15)

min=-0.399, mean=0.667, max=1.328, sum=10.009 (15)

min=-1.341, mean=-0.398, max=0.605, sum=-5.971 (15)

min=0.737, mean=0.756, max=0.786, sum=2.269 (3)

min=0.157, mean=0.182, max=0.199, sum=0.546 (3)

min=0.99, mean=0.991, max=0.992, sum=2.974 (3)

min=0.99, mean=0.997, max=1, sum=2.99 (3)

min=0.004, mean=0.009, max=0.014, sum=0.026 (3)

min=2.299, mean=2.626, max=2.886, sum=7.879 (3)

min=0.989, mean=1.05, max=1.098, sum=3.15 (3)

min=0.591, mean=0.729, max=0.908, sum=24.051 (33)

min=0.029, mean=0.178, max=0.683, sum=5.876 (33)

min=0.126, mean=0.218, max=0.683, sum=7.184 (33)

min=0.226, mean=0.826, max=0.996, sum=27.252 (33)

min=0.031, mean=0.117, max=0.232, sum=3.851 (33)

min=-0.726, mean=1.585, max=5.189, sum=52.306 (33)

min=-4.393, mean=0.105, max=3.99, sum=3.456 (33)

min=0.49, mean=0.576, max=0.71, sum=31.118 (54)

min=0.0, mean=0.303, max=0.578, sum=16.351 (54)

min=0.07, mean=0.314, max=0.578, sum=16.963 (54)

min=0.001, mean=0.556, max=1, sum=30.03 (54)

min=0, mean=0.537, max=1, sum=28.984 (54)

min=0, mean=0.037, max=0.109, sum=1.998 (54)

min=-13.279, mean=3.895, max=134.594, sum=206.455 (53)

min=-3.809, mean=0.453, max=11.336, sum=23.984 (53)

min=0.336, mean=0.484, max=0.684, sum=7.26 (15)

min=0.014, mean=0.074, max=0.134, sum=1.117 (15)

min=0.076, mean=0.134, max=0.172, sum=2.006 (15)

min=0.226, mean=0.571, max=0.921, sum=8.561 (15)

min=0.182, mean=0.69, max=1, sum=10.348 (15)

min=0.032, mean=0.097, max=0.151, sum=1.461 (15)

min=-2.049, mean=0.62, max=1.653, sum=9.301 (15)

min=-2.13, mean=-0.26, max=0.964, sum=-3.906 (15)

min=0.801, mean=0.827, max=0.846, sum=2.481 (3)

min=0.088, mean=0.111, max=0.153, sum=0.333 (3)

min=0.988, mean=0.991, max=0.994, sum=2.972 (3)

min=0.013, mean=0.015, max=0.017, sum=0.044 (3)

min=1.628, mean=1.767, max=1.953, sum=5.302 (3)

min=0.271, mean=0.547, max=0.966, sum=1.64 (3)

min=0.484, mean=0.662, max=0.831, sum=21.86 (33)