The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

mmlu

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

# eval: Number of evaluation instances.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

# train: Number of training instances (e.g., in-context examples).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

# prompt tokens: Number of tokens in the prompt.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

# output tokens: Actual number of output tokens.

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains [(Hendrycks et al., 2021)](https://openreview.net/forum?id=d7KBjmI3GmQ).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.19, mean=0.259, max=0.35, sum=3.891 (15)

min=0.074, mean=0.131, max=0.172, sum=1.96 (15)

min=0.15, mean=0.221, max=0.31, sum=3.313 (15)

min=0.17, mean=0.236, max=0.33, sum=3.545 (15)

min=0.419, mean=0.457, max=0.511, sum=6.851 (15)

min=100, mean=102.8, max=114, sum=1542 (15)

min=308.59, mean=396.74, max=552.719, sum=5951.098 (15)

min=0.2, mean=0.241, max=0.298, sum=3.617 (15)

min=0.051, mean=0.123, max=0.181, sum=1.842 (15)

min=0.16, mean=0.2, max=0.272, sum=3.002 (15)

min=0.16, mean=0.204, max=0.23, sum=3.059 (15)

min=0.348, mean=0.377, max=0.422, sum=5.648 (15)

min=0.2, mean=0.27, max=0.35, sum=4.047 (15)

min=0.063, mean=0.114, max=0.154, sum=1.708 (15)

min=0.15, mean=0.225, max=0.27, sum=3.377 (15)

min=0.158, mean=0.232, max=0.29, sum=3.474 (15)

min=0.381, mean=0.411, max=0.466, sum=6.166 (15)

min=0.23, mean=0.445, max=0.8, sum=6.677 (15)

min=0.067, mean=0.139, max=0.205, sum=2.09 (15)

min=0.2, mean=0.392, max=0.73, sum=5.887 (15)

min=0.19, mean=0.409, max=0.77, sum=6.142 (15)

min=0.23, mean=0.48, max=0.83, sum=7.207 (15)

min=0.056, mean=0.137, max=0.248, sum=2.059 (15)

min=0.17, mean=0.417, max=0.75, sum=6.251 (15)

min=0.21, mean=0.45, max=0.78, sum=6.75 (15)

min=0.24, mean=0.475, max=0.81, sum=7.13 (15)

min=0.076, mean=0.134, max=0.172, sum=2.006 (15)

min=0.22, mean=0.411, max=0.68, sum=6.168 (15)

min=0.23, mean=0.433, max=0.73, sum=6.498 (15)

min=0.211, mean=0.339, max=0.5, sum=5.078 (15)

min=0.06, mean=0.141, max=0.219, sum=2.11 (15)

min=0.17, mean=0.263, max=0.42, sum=3.938 (15)

min=0.167, mean=0.297, max=0.45, sum=4.453 (15)

min=0.193, mean=0.27, max=0.32, sum=4.045 (15)

min=0.087, mean=0.111, max=0.157, sum=1.661 (15)

min=0.1, mean=0.183, max=0.27, sum=2.74 (15)

min=0.09, mean=0.185, max=0.27, sum=2.769 (15)

min=360.75, mean=471.075, max=618.447, sum=7066.132 (15)

min=0.23, mean=0.321, max=0.49, sum=4.811 (15)

min=0.075, mean=0.135, max=0.225, sum=2.023 (15)

min=0.1, mean=0.23, max=0.37, sum=3.451 (15)

min=0.14, mean=0.237, max=0.35, sum=3.549 (15)

min=0.22, mean=0.38, max=0.61, sum=5.702 (15)

min=0.122, mean=0.154, max=0.217, sum=2.31 (15)

min=0.08, mean=0.255, max=0.51, sum=3.821 (15)

min=0.11, mean=0.264, max=0.51, sum=3.955 (15)

min=0.25, mean=0.481, max=0.78, sum=7.22 (15)

min=0.063, mean=0.144, max=0.262, sum=2.165 (15)

min=0.17, mean=0.434, max=0.76, sum=6.513 (15)

min=0.211, mean=0.447, max=0.74, sum=6.702 (15)

min=0.556, mean=0.578, max=0.605, sum=8.664 (15)

min=371.38, mean=472.274, max=624.07, sum=7084.111 (15)

min=0.19, mean=0.299, max=0.42, sum=4.481 (15)

min=0.115, mean=0.137, max=0.173, sum=2.054 (15)

min=0.167, mean=0.25, max=0.38, sum=3.754 (15)

min=0.175, mean=0.274, max=0.38, sum=4.104 (15)

min=0.135, mean=0.233, max=0.418, sum=3.493 (15)

min=333.02, mean=436.99, max=574.658, sum=6554.844 (15)

min=0.25, mean=0.407, max=0.67, sum=6.098 (15)

min=0.074, mean=0.168, max=0.3, sum=2.515 (15)

min=0.25, mean=0.378, max=0.62, sum=5.675 (15)

min=0.25, mean=0.382, max=0.63, sum=5.731 (15)

min=0.141, mean=0.145, max=0.149, sum=2.18 (15)

min=386.05, mean=492.01, max=639.561, sum=7380.154 (15)

min=0.228, mean=0.353, max=0.56, sum=5.296 (15)

min=0.089, mean=0.149, max=0.246, sum=2.242 (15)

min=0.158, mean=0.29, max=0.51, sum=4.349 (15)

min=0.158, mean=0.315, max=0.53, sum=4.729 (15)

min=0.47, mean=0.489, max=0.506, sum=7.328 (15)

min=372.75, mean=481.26, max=628.421, sum=7218.903 (15)

min=0.19, mean=0.324, max=0.4, sum=4.854 (15)

min=0.075, mean=0.112, max=0.151, sum=1.678 (15)

min=0.15, mean=0.253, max=0.35, sum=3.799 (15)

min=0.14, mean=0.281, max=0.38, sum=4.214 (15)

min=0.292, mean=0.317, max=0.349, sum=4.752 (15)

min=0.18, mean=0.279, max=0.36, sum=4.182 (15)

min=0.067, mean=0.114, max=0.164, sum=1.703 (15)

min=0.09, mean=0.184, max=0.24, sum=2.755 (15)

min=0.15, mean=0.237, max=0.29, sum=3.548 (15)

min=0.265, mean=0.281, max=0.301, sum=4.21 (15)

min=0.18, mean=0.264, max=0.42, sum=3.963 (15)

min=0.049, mean=0.136, max=0.202, sum=2.04 (15)

min=0.13, mean=0.226, max=0.42, sum=3.397 (15)

min=0.1, mean=0.222, max=0.4, sum=3.334 (15)

min=0.265, mean=0.284, max=0.312, sum=4.267 (15)

min=0.21, mean=0.382, max=0.67, sum=5.731 (15)

min=0.104, mean=0.143, max=0.197, sum=2.146 (15)

min=0.12, mean=0.299, max=0.6, sum=4.49 (15)

min=0.13, mean=0.317, max=0.57, sum=4.748 (15)

min=0.18, mean=0.254, max=0.32, sum=3.806 (15)

min=0.055, mean=0.113, max=0.167, sum=1.691 (15)

min=0.15, mean=0.207, max=0.25, sum=3.1 (15)

min=0.14, mean=0.22, max=0.3, sum=3.299 (15)

min=0.26, mean=0.406, max=0.63, sum=6.095 (15)

min=0.103, mean=0.155, max=0.243, sum=2.327 (15)

min=0.2, mean=0.334, max=0.54, sum=5.009 (15)

min=0.2, mean=0.366, max=0.55, sum=5.495 (15)

min=0.23, mean=0.452, max=0.79, sum=6.786 (15)

min=0.099, mean=0.183, max=0.338, sum=2.742 (15)

min=0.15, mean=0.387, max=0.73, sum=5.807 (15)

min=0.19, mean=0.407, max=0.73, sum=6.107 (15)

min=0.14, mean=0.249, max=0.3, sum=3.728 (15)

min=0.062, mean=0.115, max=0.149, sum=1.732 (15)

min=0.11, mean=0.217, max=0.28, sum=3.262 (15)

min=0.13, mean=0.22, max=0.27, sum=3.294 (15)

min=0.066, mean=0.07, max=0.072, sum=1.05 (15)

min=0.21, mean=0.276, max=0.351, sum=4.146 (15)

min=0.094, mean=0.122, max=0.145, sum=1.831 (15)

min=0.149, mean=0.189, max=0.24, sum=2.833 (15)

min=0.175, mean=0.215, max=0.26, sum=3.228 (15)

min=0.093, mean=0.133, max=0.275, sum=1.995 (15)

min=358.76, mean=467.936, max=612.798, sum=7019.035 (15)

min=0.211, mean=0.29, max=0.4, sum=4.354 (15)

min=0.1, mean=0.151, max=0.242, sum=2.271 (15)

min=0.19, mean=0.258, max=0.38, sum=3.866 (15)

min=0.167, mean=0.235, max=0.33, sum=3.525 (15)

min=0.173, mean=0.218, max=0.232, sum=3.277 (15)

min=2.482, mean=4.326, max=5, sum=64.896 (15)

min=382.49, mean=420.562, max=467.75, sum=6308.426 (15)

min=0.2, mean=0.291, max=0.39, sum=4.368 (15)

min=0.084, mean=0.134, max=0.202, sum=2.004 (15)

min=0.2, mean=0.272, max=0.37, sum=4.079 (15)

min=0.19, mean=0.273, max=0.36, sum=4.102 (15)

min=0.178, mean=0.182, max=0.184, sum=2.725 (15)

min=2.465, mean=4.316, max=5, sum=64.743 (15)

min=385.228, mean=423.395, max=467.79, sum=6350.919 (15)

min=0.21, mean=0.318, max=0.48, sum=4.775 (15)

min=0.115, mean=0.147, max=0.194, sum=2.207 (15)

min=0.13, mean=0.27, max=0.45, sum=4.048 (15)

min=0.167, mean=0.287, max=0.43, sum=4.298 (15)

min=0.11, mean=0.12, max=0.138, sum=1.793 (15)

min=0.2, mean=0.276, max=0.37, sum=4.141 (15)

min=0.101, mean=0.135, max=0.172, sum=2.031 (15)

min=0.13, mean=0.216, max=0.32, sum=3.242 (15)

min=0.18, mean=0.229, max=0.33, sum=3.44 (15)

min=0.041, mean=0.055, max=0.081, sum=0.818 (15)

min=0.24, mean=0.469, max=0.78, sum=7.035 (15)

min=0.073, mean=0.127, max=0.202, sum=1.908 (15)

min=0.15, mean=0.403, max=0.75, sum=6.051 (15)

min=0.17, mean=0.418, max=0.75, sum=6.266 (15)

min=0.2, mean=0.242, max=0.35, sum=3.627 (15)

min=0.103, mean=0.132, max=0.175, sum=1.983 (15)

min=0.09, mean=0.169, max=0.24, sum=2.542 (15)

min=0.17, mean=0.212, max=0.31, sum=3.186 (15)

min=0.26, mean=0.422, max=0.7, sum=6.336 (15)

min=0.093, mean=0.132, max=0.18, sum=1.976 (15)

min=0.17, mean=0.34, max=0.6, sum=5.102 (15)

min=0.24, mean=0.38, max=0.61, sum=5.705 (15)

min=0.203, mean=0.212, max=0.221, sum=3.181 (15)

min=0.19, mean=0.243, max=0.29, sum=3.642 (15)

min=0.069, mean=0.138, max=0.238, sum=2.071 (15)

min=0.1, mean=0.19, max=0.263, sum=2.854 (15)

min=0.15, mean=0.218, max=0.281, sum=3.266 (15)

min=0.091, mean=0.092, max=0.095, sum=1.387 (15)

min=0.17, mean=0.235, max=0.35, sum=3.518 (15)

min=0.095, mean=0.14, max=0.179, sum=2.093 (15)

min=0.09, mean=0.166, max=0.24, sum=2.489 (15)

min=0.14, mean=0.206, max=0.28, sum=3.085 (15)

min=0.118, mean=0.119, max=0.12, sum=1.785 (15)

min=0.132, mean=0.243, max=0.32, sum=3.641 (15)

min=0.049, mean=0.128, max=0.186, sum=1.923 (15)

min=0.105, mean=0.204, max=0.28, sum=3.054 (15)

min=0.053, mean=0.21, max=0.31, sum=3.155 (15)

min=0.14, mean=0.14, max=0.141, sum=2.103 (15)

min=0.28, mean=0.569, max=0.86, sum=8.532 (15)

min=0.127, mean=0.317, max=0.54, sum=4.761 (15)

min=0.19, mean=0.517, max=0.84, sum=7.752 (15)

min=0.24, mean=0.537, max=0.83, sum=8.054 (15)

min=0.26, mean=0.568, max=0.86, sum=8.515 (15)

min=0.064, mean=0.176, max=0.264, sum=2.644 (15)

min=0.23, mean=0.525, max=0.83, sum=7.868 (15)

min=0.24, mean=0.531, max=0.82, sum=7.964 (15)

min=0.175, mean=0.196, max=0.215, sum=2.946 (15)

min=0.21, mean=0.237, max=0.298, sum=3.558 (15)

min=0.298, mean=0.462, max=0.534, sum=6.937 (15)

min=0.16, mean=0.22, max=0.272, sum=3.303 (15)

min=0.2, mean=0.231, max=0.281, sum=3.462 (15)

min=0.129, mean=0.133, max=0.14, sum=1.998 (15)

min=0.11, mean=0.229, max=0.325, sum=3.431 (15)

min=0.16, mean=0.311, max=0.472, sum=4.659 (15)

min=0.1, mean=0.186, max=0.228, sum=2.79 (15)

min=0.09, mean=0.205, max=0.272, sum=3.077 (15)

min=0.131, mean=0.133, max=0.135, sum=1.99 (15)

min=0.14, mean=0.238, max=0.31, sum=3.566 (15)

min=0.357, mean=0.506, max=0.666, sum=7.594 (15)

min=0.08, mean=0.178, max=0.28, sum=2.665 (15)

min=0.11, mean=0.202, max=0.28, sum=3.026 (15)

min=0.086, mean=0.088, max=0.089, sum=1.314 (15)