Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

dyck_language

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

Exact match (up to specified indicator): Fraction of instances that the predicted output matches a correct reference exactly, ignoring text preceding the specified indicator.

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

# eval: Number of evaluation instances.

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

# train: Number of training instances (e.g., in-context examples).

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

# prompt tokens: Number of tokens in the prompt.

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

# output tokens: Actual number of output tokens.

Scenario testing hierarchical reasoning through the Dyck formal languages [(Suzgun et al., 2019)](https://aclanthology.org/W19-3905/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

dyck_language_num_parenthesis_pairs:3

min=0.4, mean=0.445, max=0.468, sum=1.336 (3)

min=0.438, mean=0.447, max=0.451, sum=1.341 (3)

min=206.652, mean=206.652, max=206.652, sum=619.956 (3)

min=3.218, mean=3.423, max=3.526, sum=10.27 (3)

min=0.404, mean=0.414, max=0.434, sum=1.242 (3)

min=0.347, mean=0.347, max=0.347, sum=1.04 (3)

min=4.572, mean=4.572, max=4.572, sum=13.716 (3)

min=0.668, mean=0.696, max=0.71, sum=2.088 (3)

min=0.516, mean=0.517, max=0.517, sum=1.551 (3)

min=4.008, mean=4.041, max=4.058, sum=12.124 (3)

min=0.562, mean=0.617, max=0.644, sum=1.85 (3)

min=4.366, mean=4.44, max=4.588, sum=13.32 (3)

min=0.638, mean=0.709, max=0.744, sum=2.126 (3)

min=3.52, mean=3.672, max=3.748, sum=11.016 (3)

min=0.514, mean=0.529, max=0.536, sum=1.586 (3)

min=4.714, mean=4.753, max=4.832, sum=14.26 (3)

min=0.53, mean=0.559, max=0.574, sum=1.678 (3)

min=4.586, mean=4.587, max=4.588, sum=13.76 (3)

min=0.502, mean=0.517, max=0.524, sum=1.55 (3)

min=166.418, mean=166.418, max=166.418, sum=499.254 (3)

min=2.398, mean=2.511, max=2.568, sum=7.534 (3)

min=0.654, mean=0.666, max=0.672, sum=1.998 (3)

min=2.614, mean=2.721, max=2.936, sum=8.164 (3)

min=0.686, mean=0.729, max=0.75, sum=2.186 (3)

min=3.074, mean=3.123, max=3.222, sum=9.37 (3)

min=0.846, mean=0.849, max=0.85, sum=2.546 (3)

min=0.639, mean=0.644, max=0.646, sum=1.931 (3)

min=164.418, mean=164.418, max=164.418, sum=493.254 (3)

min=2.396, mean=2.479, max=2.52, sum=7.436 (3)

min=0.528, mean=0.545, max=0.578, sum=1.634 (3)

min=0.388, mean=0.389, max=0.389, sum=1.166 (3)

min=150.706, mean=150.706, max=150.706, sum=452.118 (3)

min=0.008, mean=0.011, max=0.016, sum=0.032 (3)

min=0.166, mean=0.166, max=0.166, sum=0.498 (3)

min=265.076, mean=265.076, max=265.076, sum=795.228 (3)

min=0.542, mean=0.594, max=0.62, sum=1.782 (3)

min=0.513, mean=0.515, max=0.518, sum=1.544 (3)

min=165.418, mean=165.418, max=165.418, sum=496.254 (3)

min=3.81, mean=3.857, max=3.95, sum=11.57 (3)

min=0.366, mean=0.531, max=0.614, sum=1.594 (3)

min=0.276, mean=0.281, max=0.284, sum=0.844 (3)

min=2.186, mean=2.462, max=2.6, sum=7.386 (3)

min=0.396, mean=0.511, max=0.568, sum=1.532 (3)

min=0.245, mean=0.248, max=0.25, sum=0.745 (3)

min=3.14, mean=3.447, max=3.6, sum=10.34 (3)

min=0.354, mean=0.358, max=0.36, sum=1.074 (3)

min=0.245, mean=0.246, max=0.247, sum=0.739 (3)

min=3.016, mean=3.203, max=3.296, sum=9.608 (3)

min=0.566, mean=0.587, max=0.598, sum=1.762 (3)

min=3.68, mean=3.887, max=3.99, sum=11.66 (3)

min=0.386, mean=0.411, max=0.424, sum=1.234 (3)

min=1.834, mean=1.959, max=2.022, sum=5.878 (3)

min=0.324, mean=0.372, max=0.396, sum=1.116 (3)

min=1.562, mean=1.901, max=2.07, sum=5.702 (3)

min=0.354, mean=0.421, max=0.454, sum=1.262 (3)

min=1.448, mean=1.591, max=1.662, sum=4.772 (3)

min=0.336, mean=0.337, max=0.34, sum=1.012 (3)

min=0.312, mean=0.312, max=0.312, sum=0.936 (3)

min=0.746, mean=0.747, max=0.748, sum=2.242 (3)

min=0.325, mean=0.325, max=0.325, sum=0.974 (3)

min=0.33, mean=0.347, max=0.356, sum=1.042 (3)

min=0.234, mean=0.235, max=0.237, sum=0.705 (3)

min=0.12, mean=0.14, max=0.15, sum=0.42 (3)

min=0.216, mean=0.216, max=0.216, sum=0.647 (3)

min=269.076, mean=269.076, max=269.076, sum=807.228 (3)

min=0.45, mean=0.494, max=0.516, sum=1.482 (3)

min=1.251, mean=1.256, max=1.259, sum=3.768 (3)

min=0.436, mean=0.471, max=0.488, sum=1.412 (3)

min=0.054, mean=0.054, max=0.055, sum=0.163 (3)

min=0.702, mean=0.753, max=0.778, sum=2.258 (3)

min=2.176, mean=2.252, max=2.29, sum=6.756 (3)

min=0.536, mean=0.554, max=0.59, sum=1.662 (3)

min=3.068, mean=3.352, max=3.494, sum=10.056 (3)

min=0.612, mean=0.668, max=0.696, sum=2.004 (3)

min=0.233, mean=0.235, max=0.236, sum=0.705 (3)

min=1.842, mean=1.946, max=1.998, sum=5.838 (3)

min=0.414, mean=0.47, max=0.498, sum=1.41 (3)

min=0.093, mean=0.094, max=0.094, sum=0.281 (3)

min=1.356, mean=1.479, max=1.54, sum=4.436 (3)

min=0.366, mean=0.473, max=0.526, sum=1.418 (3)

min=0.126, mean=0.128, max=0.131, sum=0.383 (3)

min=2.656, mean=2.977, max=3.62, sum=8.932 (3)

min=0.392, mean=0.406, max=0.434, sum=1.218 (3)

min=0.148, mean=0.149, max=0.15, sum=0.447 (3)

min=2.232, mean=2.335, max=2.54, sum=7.004 (3)

min=0.732, mean=0.751, max=0.76, sum=2.252 (3)

min=2.268, mean=2.297, max=2.354, sum=6.89 (3)

min=0.562, mean=0.603, max=0.624, sum=1.81 (3)

min=0.207, mean=0.208, max=0.211, sum=0.625 (3)

min=1.58, mean=1.637, max=1.75, sum=4.91 (3)

min=0.378, mean=0.41, max=0.426, sum=1.23 (3)

min=0.136, mean=0.136, max=0.137, sum=0.408 (3)

min=1.378, mean=1.436, max=1.552, sum=4.308 (3)

min=0.208, mean=0.231, max=0.278, sum=0.694 (3)

min=0.133, mean=0.134, max=0.134, sum=0.401 (3)

min=1.372, mean=1.472, max=1.672, sum=4.416 (3)

min=0.158, mean=0.161, max=0.168, sum=0.484 (3)

min=0.089, mean=0.09, max=0.091, sum=0.27 (3)

min=1.568, mean=1.761, max=1.858, sum=5.284 (3)

min=0.772, mean=0.805, max=0.822, sum=2.416 (3)

min=0.222, mean=0.223, max=0.225, sum=0.669 (3)

min=3.228, mean=3.267, max=3.346, sum=9.802 (3)

min=0.396, mean=0.451, max=0.478, sum=1.352 (3)

min=0.107, mean=0.107, max=0.107, sum=0.321 (3)

min=2.592, mean=2.593, max=2.594, sum=7.78 (3)

min=0.172, mean=0.204, max=0.22, sum=0.612 (3)

min=159.418, mean=159.418, max=159.418, sum=478.254 (3)

min=5.458, mean=5.527, max=5.666, sum=16.582 (3)

min=0.53, mean=0.53, max=0.53, sum=0.53 (1)

min=164.418, mean=164.418, max=164.418, sum=164.418 (1)

min=0.536, mean=0.549, max=0.574, sum=1.646 (3)

min=1.215, mean=1.216, max=1.216, sum=3.647 (3)

min=203.784, mean=203.784, max=203.784, sum=611.352 (3)

min=4.446, mean=4.583, max=4.652, sum=13.75 (3)

min=0.39, mean=0.447, max=0.476, sum=1.342 (3)

min=1.004, mean=1.045, max=1.066, sum=3.136 (3)

min=0.542, mean=0.579, max=0.598, sum=1.738 (3)

min=2.636, mean=2.711, max=2.748, sum=8.132 (3)

min=0.628, mean=0.633, max=0.644, sum=1.9 (3)

min=0.094, mean=0.273, max=0.631, sum=0.82 (3)