Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

synthetic_reasoning

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# eval: Number of evaluation instances.

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# train: Number of training instances (e.g., in-context examples).

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# prompt tokens: Number of tokens in the prompt.

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# output tokens: Actual number of output tokens.

Synthetic reasoning tasks defined using abstract symbols based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.043, mean=0.263, max=0.46, sum=2.363 (9)

min=0.543, mean=0.589, max=0.705, sum=8.84 (15)

min=515, mean=515, max=515, sum=7725 (15)

min=153.775, mean=246.737, max=322.217, sum=3701.054 (15)

min=4.412, mean=6.116, max=9.454, sum=91.742 (15)

min=0.029, mean=0.201, max=0.328, sum=1.808 (9)

min=0.368, mean=0.399, max=0.432, sum=5.985 (15)

min=5.476, mean=6.371, max=9.744, sum=95.567 (15)

min=0.041, mean=0.247, max=0.437, sum=2.225 (9)

min=0.474, mean=0.531, max=0.632, sum=7.96 (15)

min=5.361, mean=6.171, max=8.983, sum=92.567 (15)

min=0.037, mean=0.286, max=0.627, sum=2.575 (9)

min=5.291, mean=6.146, max=8.208, sum=92.184 (15)

min=0.064, mean=0.393, max=0.767, sum=3.538 (9)

min=5.295, mean=6.232, max=7.775, sum=93.476 (15)

min=0.031, mean=0.301, max=0.6, sum=2.709 (9)

min=5.47, mean=6.303, max=8.792, sum=94.551 (15)

min=0.041, mean=0.192, max=0.379, sum=1.724 (9)

min=3.441, mean=5.819, max=8.951, sum=87.287 (15)

min=0.043, mean=0.209, max=0.449, sum=1.88 (9)

min=170.128, mean=294.059, max=427.445, sum=4410.88 (15)

min=0, mean=3.593, max=9.732, sum=53.897 (15)

min=0.027, mean=0.225, max=0.394, sum=2.027 (9)

min=0, mean=3.645, max=9.045, sum=54.682 (15)

min=0.049, mean=0.312, max=0.52, sum=2.81 (9)

min=0, mean=3.797, max=9.614, sum=56.961 (15)

min=0.047, mean=0.432, max=0.781, sum=3.891 (9)

min=0.806, mean=0.868, max=1.072, sum=13.014 (15)

min=170.435, mean=286.053, max=411.15, sum=4290.796 (15)

min=4.557, mean=5.829, max=9.223, sum=87.431 (15)

min=0.054, mean=0.304, max=0.524, sum=2.74 (9)

min=0.272, mean=0.34, max=0.505, sum=5.1 (15)

min=161.472, mean=274.195, max=391.014, sum=4112.924 (15)

min=9, mean=12.028, max=19.262, sum=180.419 (15)

min=0.262, mean=0.296, max=0.356, sum=4.443 (15)

min=202.478, mean=318.235, max=432.755, sum=4773.52 (15)

min=0.035, mean=0.194, max=0.406, sum=1.746 (9)

min=0.477, mean=0.589, max=0.939, sum=8.832 (15)

min=172.105, mean=292.607, max=424.033, sum=4389.105 (15)

min=0, mean=4.073, max=13.522, sum=61.095 (15)

min=0.029, mean=0.128, max=0.231, sum=1.155 (9)

min=0.296, mean=0.353, max=0.561, sum=5.291 (15)

min=0, mean=3.927, max=13.843, sum=58.901 (15)

min=0.002, mean=0.129, max=0.278, sum=1.157 (9)

min=0.268, mean=0.284, max=0.348, sum=4.262 (15)

min=0, mean=3.753, max=10.59, sum=56.301 (15)

min=0.035, mean=0.121, max=0.239, sum=1.089 (9)

min=0.258, mean=0.269, max=0.293, sum=4.03 (15)

min=0, mean=3.558, max=9.711, sum=53.377 (15)

min=0.035, mean=0.229, max=0.4, sum=2.058 (9)

min=0, mean=3.876, max=10.584, sum=58.144 (15)

min=0.033, mean=0.096, max=0.198, sum=0.866 (9)

min=0, mean=3.716, max=9.66, sum=55.736 (15)

min=0.031, mean=0.123, max=0.204, sum=1.111 (9)

min=4.07, mean=5.674, max=10.377, sum=85.109 (15)

min=0.047, mean=0.243, max=0.423, sum=2.19 (9)

min=4.379, mean=5.706, max=9.528, sum=85.592 (15)

min=0.047, mean=0.174, max=0.363, sum=1.565 (9)

min=0.076, mean=0.136, max=0.266, sum=2.039 (15)

min=9, mean=17.917, max=39.722, sum=268.757 (15)

min=0.033, mean=0.204, max=0.386, sum=1.835 (9)

min=0.095, mean=0.14, max=0.277, sum=2.107 (15)

min=169.79, mean=287.815, max=418.918, sum=4317.223 (15)

min=9, mean=19.93, max=50, sum=298.944 (15)

min=0.033, mean=0.196, max=0.404, sum=1.763 (9)

min=0.44, mean=0.531, max=0.625, sum=7.971 (15)

min=202.478, mean=317.835, max=431.755, sum=4767.52 (15)

min=0.049, mean=0.205, max=0.464, sum=1.847 (9)

min=0.341, mean=0.387, max=0.455, sum=5.808 (15)

min=206.478, mean=321.835, max=435.755, sum=4827.52 (15)

min=0.008, mean=0.225, max=0.441, sum=2.029 (9)

min=0.179, mean=0.257, max=0.411, sum=3.858 (15)

min=9, mean=13.045, max=24.32, sum=195.68 (15)

min=0.033, mean=0.193, max=0.394, sum=1.734 (9)

min=0.054, mean=0.075, max=0.157, sum=1.127 (15)

min=9, mean=13.868, max=40.338, sum=208.016 (15)

min=0.064, mean=0.362, max=0.701, sum=3.262 (9)

min=3.738, mean=5.457, max=7.707, sum=81.852 (15)

min=0.039, mean=0.232, max=0.462, sum=2.091 (9)

min=4.528, mean=5.977, max=11.827, sum=89.662 (15)

min=0.045, mean=0.236, max=0.406, sum=2.122 (9)

min=0.311, mean=0.348, max=0.449, sum=5.218 (15)

min=4.326, mean=5.535, max=8.981, sum=83.027 (15)

min=0.025, mean=0.223, max=0.472, sum=2.01 (9)

min=0.117, mean=0.124, max=0.151, sum=1.858 (15)

min=4.623, mean=5.709, max=9.526, sum=85.635 (15)

min=0.033, mean=0.16, max=0.322, sum=1.439 (9)

min=0.141, mean=0.152, max=0.208, sum=2.279 (15)

min=4.862, mean=6.671, max=16.113, sum=100.07 (15)

min=0.027, mean=0.1, max=0.181, sum=0.897 (9)

min=0.161, mean=0.173, max=0.215, sum=2.602 (15)

min=4.157, mean=6.435, max=13.249, sum=96.518 (15)

min=0.097, mean=0.502, max=0.983, sum=4.522 (9)

min=4.466, mean=5.846, max=7.742, sum=87.685 (15)

min=0.085, mean=0.488, max=0.946, sum=4.39 (9)

min=0.296, mean=0.339, max=0.388, sum=5.086 (15)

min=4.468, mean=5.85, max=7.707, sum=87.753 (15)

min=0.029, mean=0.19, max=0.344, sum=1.709 (9)

min=0.157, mean=0.165, max=0.19, sum=2.473 (15)

min=4.045, mean=5.659, max=8.903, sum=84.883 (15)

min=0.033, mean=0.123, max=0.243, sum=1.107 (9)

min=0.146, mean=0.161, max=0.174, sum=2.419 (15)

min=3.328, mean=5.496, max=7.371, sum=82.445 (15)

min=0.027, mean=0.063, max=0.122, sum=0.569 (9)

min=0.101, mean=0.113, max=0.121, sum=1.694 (15)

min=3.414, mean=5.523, max=6.753, sum=82.841 (15)

min=0.109, mean=0.54, max=0.99, sum=4.856 (9)

min=0.156, mean=0.271, max=0.382, sum=4.058 (15)

min=0, mean=4.509, max=8.757, sum=67.631 (15)

min=0.045, mean=0.341, max=0.67, sum=3.066 (9)

min=0.098, mean=0.129, max=0.171, sum=1.934 (15)

min=0, mean=4.389, max=9.806, sum=65.829 (15)

min=0.047, mean=0.455, max=0.948, sum=4.091 (9)

min=156.546, mean=259.474, max=372.394, sum=3892.105 (15)

min=4.402, mean=6.403, max=8.309, sum=96.049 (15)

min=0.035, mean=0.154, max=0.326, sum=0.462 (3)

min=170.79, mean=281.682, max=392.173, sum=1408.408 (5)

min=0.039, mean=0.252, max=0.464, sum=2.264 (9)

min=0.226, mean=0.347, max=0.624, sum=5.199 (15)

min=160.41, mean=268.083, max=380.468, sum=4021.247 (15)

min=5.792, mean=6.619, max=9.344, sum=99.282 (15)

min=0.035, mean=0.231, max=0.449, sum=2.082 (9)

min=4.27, mean=5.423, max=8.227, sum=81.338 (15)

min=0.078, mean=0.504, max=0.915, sum=4.532 (9)

min=4.344, mean=5.88, max=7.835, sum=88.2 (15)

min=0, mean=0.056, max=0.105, sum=0.507 (9)

min=0.18, mean=0.307, max=0.424, sum=4.603 (15)

min=166.074, mean=285.977, max=416.606, sum=4289.654 (15)

min=13.353, mean=29.404, max=45.429, sum=441.06 (15)

Synthetic reasoning (abstract symbols)

synthetic_reasoning_mode:induction

min=0.043, mean=0.048, max=0.054, sum=0.144 (3)

min=0.58, mean=0.584, max=0.588, sum=1.752 (3)

min=153.775, mean=154.108, max=154.775, sum=462.324 (3)

min=7.12, mean=7.223, max=7.32, sum=21.668 (3)

min=0.029, mean=0.033, max=0.037, sum=0.099 (3)

min=0.374, mean=0.377, max=0.38, sum=1.13 (3)

min=7.146, mean=7.28, max=7.47, sum=21.841 (3)

min=0.041, mean=0.048, max=0.052, sum=0.144 (3)

min=0.534, mean=0.534, max=0.535, sum=1.603 (3)

min=6.955, mean=6.971, max=6.988, sum=20.913 (3)

min=0.037, mean=0.04, max=0.045, sum=0.12 (3)

min=6.773, mean=7.021, max=7.278, sum=21.064 (3)

min=0.064, mean=0.066, max=0.07, sum=0.198 (3)

min=7.332, mean=7.413, max=7.483, sum=22.239 (3)

min=0.031, mean=0.038, max=0.041, sum=0.113 (3)

min=7.006, mean=7.328, max=7.534, sum=21.984 (3)

min=0.041, mean=0.044, max=0.049, sum=0.132 (3)

min=6.816, mean=6.946, max=7.138, sum=20.839 (3)

min=0.043, mean=0.045, max=0.049, sum=0.136 (3)

min=170.128, mean=173.128, max=175.128, sum=519.384 (3)

min=5.759, mean=5.862, max=5.977, sum=17.586 (3)

min=0.027, mean=0.044, max=0.056, sum=0.132 (3)

min=6.004, mean=6.203, max=6.39, sum=18.61 (3)

min=0.049, mean=0.052, max=0.058, sum=0.157 (3)

min=6.128, mean=6.357, max=6.594, sum=19.07 (3)

min=0.047, mean=0.052, max=0.06, sum=0.155 (3)

min=0.854, mean=0.862, max=0.871, sum=2.585 (3)

min=170.435, mean=172.435, max=175.435, sum=517.305 (3)

min=6.097, mean=6.201, max=6.384, sum=18.604 (3)

min=0.054, mean=0.061, max=0.068, sum=0.184 (3)

min=0.325, mean=0.372, max=0.417, sum=1.115 (3)

min=161.472, mean=163.472, max=165.472, sum=490.416 (3)

min=12.72, mean=14.279, max=16.732, sum=42.837 (3)

min=0.27, mean=0.288, max=0.326, sum=0.865 (3)

min=202.478, mean=205.478, max=209.478, sum=616.433 (3)

min=0.035, mean=0.042, max=0.05, sum=0.126 (3)

min=0.603, mean=0.613, max=0.624, sum=1.84 (3)

min=172.105, mean=172.772, max=174.105, sum=518.315 (3)

min=5.975, mean=6.228, max=6.483, sum=18.685 (3)

min=0.029, mean=0.031, max=0.033, sum=0.093 (3)

min=0.362, mean=0.37, max=0.375, sum=1.11 (3)

min=5.965, mean=6.311, max=6.553, sum=18.934 (3)

min=0.002, mean=0.023, max=0.037, sum=0.07 (3)

min=0.276, mean=0.278, max=0.28, sum=0.833 (3)

min=5.612, mean=5.759, max=5.977, sum=17.278 (3)

min=0.035, mean=0.039, max=0.043, sum=0.118 (3)

min=0.258, mean=0.26, max=0.263, sum=0.781 (3)

min=5.192, mean=5.507, max=6.006, sum=16.522 (3)

min=0.035, mean=0.047, max=0.056, sum=0.14 (3)

min=6.016, mean=6.228, max=6.46, sum=18.685 (3)

min=0.033, mean=0.041, max=0.049, sum=0.124 (3)

min=5.534, mean=6.028, max=6.567, sum=18.083 (3)

min=0.031, mean=0.047, max=0.058, sum=0.142 (3)

min=5.46, mean=5.768, max=6.087, sum=17.303 (3)

min=0.047, mean=0.05, max=0.056, sum=0.151 (3)

min=5.767, mean=5.874, max=6.019, sum=17.621 (3)

min=0.047, mean=0.049, max=0.05, sum=0.146 (3)

min=0.076, mean=0.089, max=0.104, sum=0.266 (3)

min=9.753, mean=11.026, max=13.198, sum=33.078 (3)

min=0.033, mean=0.043, max=0.05, sum=0.13 (3)