Targeted evaluation of reasoning capabilities (e.g. mathematical, hierarchical).

reasoning

How many models this model outperform on average (over columns).

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

F1 (set match): Average F1 score in terms of set overlap between the model predicted set and correct reference set.

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

Exact match (up to specified indicator): Fraction of instances that the predicted output matches a correct reference exactly, ignoring text preceding the specified indicator.

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

Exact match (up to specified indicator): Fraction of instances that the predicted output matches a correct reference exactly, ignoring text preceding the specified indicator.

The MATH benchmark for measuring mathematical problem solving on competition math problems [(Hendrycks et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html).

Equivalent: Fraction of model outputs that are mathematically equivalent to the correct reference.

The MATH benchmark for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning [(Hendrycks et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html).

Equivalent (chain of thought): Fraction of model outputs that are mathematically equivalent to the correct reference when using chain-of-thoughts prompting.

The HumanEval benchmark for measuring functional correctness for synthesizing programs from docstrings [(Chen et al., 2021)](https://arxiv.org/pdf/2107.03374.pdf).

pass@1: Fraction of model outputs that pass the associated test cases.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Scenario from [Mei et al. (2021)](https://ieeexplore.ieee.org/document/9458712/) that tests the ability to impute missing entities in a data table.

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Scenario from Magellan [(Konda et al., 2016)](https://dl.acm.org/doi/10.14778/3007263.3007314) that tests the ability to determine if two entities match.

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

accuracy

min=0.043, mean=0.263, max=0.46, sum=2.363 (9)

min=0.12, mean=0.174, max=0.239, sum=2.093 (12)

min=0.506, mean=0.543, max=0.568, sum=1.63 (3)

min=0.4, mean=0.445, max=0.468, sum=1.336 (3)

min=0.051, mean=0.054, max=0.059, sum=0.162 (3)

min=0.038, mean=0.089, max=0.14, sum=1.86 (21)

min=0, mean=0.033, max=0.105, sum=0.7 (21)

min=0.217, mean=0.232, max=0.248, sum=0.696 (3)

min=0.483, mean=0.484, max=0.485, sum=1.452 (3)

min=0.593, mean=0.735, max=0.877, sum=4.41 (6)

min=0.752, mean=0.841, max=0.934, sum=7.573 (9)

min=0.029, mean=0.201, max=0.328, sum=1.808 (9)

min=0.126, mean=0.154, max=0.183, sum=1.845 (12)

min=0.439, mean=0.469, max=0.485, sum=1.407 (3)

min=0.404, mean=0.414, max=0.434, sum=1.242 (3)

min=0.013, mean=0.014, max=0.015, sum=0.041 (3)

min=0.015, mean=0.049, max=0.133, sum=1.029 (21)

min=0, mean=0.031, max=0.14, sum=0.645 (21)

min=0.183, mean=0.196, max=0.204, sum=0.587 (3)

min=0.503, mean=0.514, max=0.532, sum=1.542 (3)

min=0.581, mean=0.729, max=0.846, sum=4.372 (6)

min=0.734, mean=0.827, max=0.89, sum=7.446 (9)

min=0.041, mean=0.247, max=0.437, sum=2.225 (9)

min=0.102, mean=0.154, max=0.234, sum=1.852 (12)

min=0.43, mean=0.458, max=0.5, sum=1.373 (3)

min=0.668, mean=0.696, max=0.71, sum=2.088 (3)

min=0.049, mean=0.054, max=0.059, sum=0.161 (3)

min=0.018, mean=0.08, max=0.167, sum=1.672 (21)

min=0, mean=0.045, max=0.133, sum=0.941 (21)

min=0.178, mean=0.188, max=0.2, sum=0.565 (3)

min=0.481, mean=0.504, max=0.544, sum=1.513 (3)

min=0.616, mean=0.729, max=0.831, sum=4.376 (6)

min=0.752, mean=0.831, max=0.88, sum=7.477 (9)

min=0.037, mean=0.286, max=0.627, sum=2.575 (9)

min=0.06, mean=0.139, max=0.204, sum=1.667 (12)

min=0.428, mean=0.47, max=0.511, sum=1.409 (3)

min=0.562, mean=0.617, max=0.644, sum=1.85 (3)

min=0.082, mean=0.096, max=0.109, sum=0.288 (3)

min=0.026, mean=0.127, max=0.205, sum=2.675 (21)

min=0, mean=0.068, max=0.135, sum=1.42 (21)

min=0.183, mean=0.191, max=0.204, sum=0.574 (3)

min=0.55, mean=0.562, max=0.573, sum=1.687 (3)

min=0.721, mean=0.8, max=0.862, sum=4.798 (6)

min=0.761, mean=0.844, max=0.901, sum=7.599 (9)

min=0.064, mean=0.393, max=0.767, sum=3.538 (9)

min=0.058, mean=0.144, max=0.245, sum=1.731 (12)

min=0.493, mean=0.521, max=0.536, sum=1.564 (3)

min=0.638, mean=0.709, max=0.744, sum=2.126 (3)

min=0.196, mean=0.225, max=0.25, sum=0.674 (3)

min=0.077, mean=0.196, max=0.316, sum=4.106 (21)

min=0.033, mean=0.086, max=0.193, sum=1.801 (21)

min=0.204, mean=0.219, max=0.239, sum=0.657 (3)

min=0.62, mean=0.639, max=0.656, sum=1.916 (3)

min=0.814, mean=0.841, max=0.877, sum=5.046 (6)

min=0.367, mean=0.79, max=0.923, sum=7.114 (9)

min=0.031, mean=0.301, max=0.6, sum=2.709 (9)

min=0.111, mean=0.164, max=0.231, sum=1.965 (12)

min=0.44, mean=0.455, max=0.479, sum=1.364 (3)

min=0.514, mean=0.529, max=0.536, sum=1.586 (3)

min=0.107, mean=0.133, max=0.15, sum=0.399 (3)

min=0.081, mean=0.146, max=0.2, sum=3.075 (21)

min=0, mean=0.054, max=0.175, sum=1.141 (21)

min=0.217, mean=0.22, max=0.226, sum=0.661 (3)

min=0.573, mean=0.575, max=0.577, sum=1.724 (3)

min=0.802, mean=0.829, max=0.862, sum=4.976 (6)

min=0.752, mean=0.849, max=0.945, sum=7.64 (9)

min=0.041, mean=0.192, max=0.379, sum=1.724 (9)

min=0.142, mean=0.176, max=0.199, sum=2.114 (12)

min=0.481, mean=0.504, max=0.528, sum=1.512 (3)

min=0.53, mean=0.559, max=0.574, sum=1.678 (3)

min=0.02, mean=0.03, max=0.045, sum=0.09 (3)

min=0, mean=0.07, max=0.233, sum=1.477 (21)

min=0, mean=0.023, max=0.088, sum=0.487 (21)

min=0.204, mean=0.217, max=0.235, sum=0.652 (3)

min=0.548, mean=0.558, max=0.569, sum=1.675 (3)

min=0.581, mean=0.731, max=0.892, sum=4.387 (6)

min=0.752, mean=0.836, max=0.88, sum=7.521 (9)

min=0.043, mean=0.209, max=0.449, sum=1.88 (9)

min=0.426, mean=0.452, max=0.487, sum=1.355 (3)

min=0.502, mean=0.517, max=0.524, sum=1.55 (3)

min=0.021, mean=0.026, max=0.029, sum=0.078 (3)

min=0.018, mean=0.089, max=0.267, sum=1.859 (21)

min=0, mean=0.026, max=0.088, sum=0.537 (21)

min=0.2, mean=0.235, max=0.261, sum=0.704 (3)

min=0.503, mean=0.513, max=0.519, sum=1.54 (3)

min=0.581, mean=0.725, max=0.862, sum=4.352 (6)

min=0.303, mean=0.543, max=0.726, sum=4.888 (9)

min=0.027, mean=0.225, max=0.394, sum=2.027 (9)

min=0.446, mean=0.475, max=0.491, sum=1.424 (3)

min=0.654, mean=0.666, max=0.672, sum=1.998 (3)

min=0.064, mean=0.067, max=0.071, sum=0.2 (3)

min=0.052, mean=0.111, max=0.167, sum=2.333 (21)

min=0, mean=0.035, max=0.088, sum=0.736 (21)

min=0.187, mean=0.188, max=0.191, sum=0.565 (3)

min=0.495, mean=0.517, max=0.534, sum=1.55 (3)

min=0.57, mean=0.722, max=0.846, sum=4.333 (6)

min=0.248, mean=0.635, max=0.912, sum=5.711 (9)

min=0.049, mean=0.312, max=0.52, sum=2.81 (9)

min=0.471, mean=0.504, max=0.522, sum=1.511 (3)

min=0.686, mean=0.729, max=0.75, sum=2.186 (3)

min=0.096, mean=0.112, max=0.12, sum=0.336 (3)

min=0.067, mean=0.149, max=0.237, sum=3.133 (21)

min=0, mean=0.057, max=0.158, sum=1.206 (21)

min=0.204, mean=0.212, max=0.222, sum=0.635 (3)

min=0.517, mean=0.53, max=0.548, sum=1.589 (3)

min=0.628, mean=0.758, max=0.831, sum=4.55 (6)

min=0.523, mean=0.707, max=0.912, sum=6.363 (9)

min=0.047, mean=0.432, max=0.781, sum=3.891 (9)

min=0.133, mean=0.259, max=0.396, sum=3.108 (12)

min=0.425, mean=0.461, max=0.508, sum=1.382 (3)

min=0.846, mean=0.849, max=0.85, sum=2.546 (3)

min=0.151, mean=0.171, max=0.182, sum=0.513 (3)

min=0.096, mean=0.198, max=0.308, sum=4.157 (21)

min=0.1, mean=0.162, max=0.244, sum=3.403 (21)

min=0.2, mean=0.213, max=0.23, sum=0.639 (3)

min=0.622, mean=0.624, max=0.628, sum=1.873 (3)

min=0.57, mean=0.733, max=0.892, sum=4.398 (6)

min=0.237, mean=0.71, max=0.923, sum=6.388 (9)

min=0.054, mean=0.304, max=0.524, sum=2.74 (9)

min=0.115, mean=0.197, max=0.3, sum=2.364 (12)

min=0.409, mean=0.447, max=0.475, sum=1.34 (3)

min=0.528, mean=0.545, max=0.578, sum=1.634 (3)

min=0.087, mean=0.095, max=0.102, sum=0.285 (3)

min=0, mean=0.043, max=0.105, sum=0.913 (21)

min=0, mean=0.055, max=0.115, sum=1.154 (21)

min=0.2, mean=0.209, max=0.217, sum=0.626 (3)

min=0.515, mean=0.543, max=0.558, sum=1.628 (3)

min=0.5, mean=0.677, max=0.831, sum=4.062 (6)

min=0.752, mean=0.852, max=0.897, sum=7.669 (9)

min=0, mean=0.002, max=0.006, sum=0.023 (12)

min=0.008, mean=0.011, max=0.016, sum=0.032 (3)

min=0.174, mean=0.186, max=0.204, sum=0.557 (3)

min=0.597, mean=0.611, max=0.626, sum=1.834 (3)

min=0, mean=0.004, max=0.012, sum=0.023 (6)

min=0.035, mean=0.194, max=0.406, sum=1.746 (9)

min=0.472, mean=0.486, max=0.498, sum=1.458 (3)

min=0.542, mean=0.594, max=0.62, sum=1.782 (3)

min=0.06, mean=0.07, max=0.076, sum=0.21 (3)

min=0.026, mean=0.135, max=0.233, sum=2.843 (21)

min=0, mean=0.054, max=0.193, sum=1.141 (21)

min=0.191, mean=0.2, max=0.213, sum=0.6 (3)

min=0.546, mean=0.558, max=0.573, sum=1.675 (3)

min=0.651, mean=0.785, max=0.846, sum=4.713 (6)

min=0.679, mean=0.823, max=0.874, sum=7.404 (9)

min=0.029, mean=0.128, max=0.231, sum=1.155 (9)

min=0.322, mean=0.36, max=0.38, sum=1.081 (3)

min=0.366, mean=0.531, max=0.614, sum=1.594 (3)

min=0.015, mean=0.018, max=0.019, sum=0.053 (3)

min=0, mean=0.073, max=0.151, sum=1.542 (21)

min=0, mean=0.035, max=0.133, sum=0.744 (21)

min=0.183, mean=0.193, max=0.209, sum=0.578 (3)

min=0.479, mean=0.491, max=0.511, sum=1.474 (3)

min=0.581, mean=0.71, max=0.831, sum=4.26 (6)

min=0.385, mean=0.713, max=0.872, sum=6.416 (9)

min=0.002, mean=0.129, max=0.278, sum=1.157 (9)

min=0.383, mean=0.391, max=0.395, sum=1.172 (3)

min=0.396, mean=0.511, max=0.568, sum=1.532 (3)

min=0.009, mean=0.015, max=0.021, sum=0.044 (3)

min=0, mean=0.049, max=0.167, sum=1.027 (21)

min=0, mean=0.027, max=0.105, sum=0.565 (21)

min=0.204, mean=0.212, max=0.226, sum=0.635 (3)

min=0.501, mean=0.507, max=0.517, sum=1.521 (3)

min=0.581, mean=0.721, max=0.862, sum=4.329 (6)

min=0.128, mean=0.482, max=0.846, sum=4.334 (9)

min=0.035, mean=0.121, max=0.239, sum=1.089 (9)

min=0.291, mean=0.306, max=0.332, sum=0.918 (3)

min=0.354, mean=0.358, max=0.36, sum=1.074 (3)

min=0.001, mean=0.004, max=0.009, sum=0.012 (3)

min=0, mean=0.016, max=0.1, sum=0.33 (21)

min=0, mean=0.003, max=0.035, sum=0.053 (21)

min=0.183, mean=0.187, max=0.196, sum=0.561 (3)

min=0.513, mean=0.524, max=0.536, sum=1.571 (3)

min=0.035, mean=0.47, max=0.846, sum=2.818 (6)

min=0.126, mean=0.176, max=0.248, sum=1.582 (9)

min=0.035, mean=0.229, max=0.4, sum=2.058 (9)

min=0.422, mean=0.439, max=0.464, sum=1.318 (3)

min=0.566, mean=0.587, max=0.598, sum=1.762 (3)

min=0.083, mean=0.1, max=0.111, sum=0.299 (3)

min=0.053, mean=0.132, max=0.267, sum=2.767 (21)

min=0, mean=0.063, max=0.158, sum=1.329 (21)

min=0.191, mean=0.204, max=0.222, sum=0.613 (3)

min=0.493, mean=0.526, max=0.556, sum=1.577 (3)

min=0.698, mean=0.803, max=0.877, sum=4.821 (6)

min=0.486, mean=0.812, max=0.934, sum=7.307 (9)

min=0.033, mean=0.096, max=0.198, sum=0.866 (9)