The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

Exact match (up to specified indicator): Fraction of instances that the predicted output matches a correct reference exactly, ignoring text preceding the specified indicator.

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

# eval: Number of evaluation instances.

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

# train: Number of training instances (e.g., in-context examples).

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

# prompt tokens: Number of tokens in the prompt.

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

# output tokens: Actual number of output tokens.

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems [(Cobbe et al., 2021)](https://arxiv.org/pdf/2110.14168.pdf).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

gsm_

min=0.051, mean=0.054, max=0.059, sum=0.162 (3)

min=7.095, mean=7.243, max=7.376, sum=21.728 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=715.394, mean=789.727, max=830.394, sum=2369.182 (3)

min=156.11, mean=158.822, max=161.567, sum=476.466 (3)

min=0.013, mean=0.014, max=0.015, sum=0.041 (3)

min=3.643, mean=3.907, max=4.168, sum=11.722 (3)

min=196.613, mean=214.018, max=229.587, sum=642.053 (3)

min=0.049, mean=0.054, max=0.059, sum=0.161 (3)

min=3.506, mean=3.72, max=3.915, sum=11.161 (3)

min=143.08, mean=152.079, max=160.784, sum=456.238 (3)

min=0.082, mean=0.096, max=0.109, sum=0.288 (3)

min=131.705, mean=136.418, max=143.462, sum=409.255 (3)

min=0.196, mean=0.225, max=0.25, sum=0.674 (3)

min=102.036, mean=111.793, max=126.641, sum=335.379 (3)

min=0.107, mean=0.133, max=0.15, sum=0.399 (3)

min=121.336, mean=126.368, max=133.267, sum=379.103 (3)

min=0.02, mean=0.03, max=0.045, sum=0.09 (3)

min=196.446, mean=208.932, max=215.567, sum=626.797 (3)

min=0.021, mean=0.026, max=0.029, sum=0.078 (3)

min=779.121, mean=877.454, max=943.121, sum=2632.363 (3)

min=0.064, mean=0.067, max=0.071, sum=0.2 (3)

min=0.096, mean=0.112, max=0.12, sum=0.336 (3)

min=0.151, mean=0.171, max=0.182, sum=0.513 (3)

min=6.037, mean=6.171, max=6.262, sum=18.514 (3)

min=785.869, mean=880.202, max=938.869, sum=2640.607 (3)

min=96.155, mean=98.636, max=100.536, sum=295.909 (3)

min=0.087, mean=0.095, max=0.102, sum=0.285 (3)

min=8.69, mean=9.403, max=9.778, sum=28.208 (3)

min=748.412, mean=823.745, max=874.412, sum=2471.236 (3)

min=332.156, mean=363.933, max=379.872, sum=1091.798 (3)

min=4.378, mean=4.584, max=4.799, sum=13.753 (3)

min=4.787, mean=4.927, max=5, sum=14.782 (3)

min=841.166, mean=927.104, max=993.696, sum=2781.313 (3)

min=0.06, mean=0.07, max=0.076, sum=0.21 (3)

min=6.446, mean=6.722, max=7.02, sum=20.166 (3)

min=783.424, mean=883.757, max=942.424, sum=2651.272 (3)

min=147.197, mean=154.41, max=162.637, sum=463.229 (3)

min=0.015, mean=0.018, max=0.019, sum=0.053 (3)

min=3.766, mean=4.515, max=5.023, sum=13.544 (3)

min=147.951, mean=179.471, max=201.126, sum=538.413 (3)

min=0.009, mean=0.015, max=0.021, sum=0.044 (3)

min=1.7, mean=2.315, max=3.133, sum=6.946 (3)

min=115.271, mean=164.999, max=232.472, sum=494.997 (3)

min=0.001, mean=0.004, max=0.009, sum=0.012 (3)

min=1.658, mean=1.796, max=1.934, sum=5.388 (3)

min=217.061, mean=241.523, max=263.371, sum=724.57 (3)

min=0.083, mean=0.1, max=0.111, sum=0.299 (3)

min=127.439, mean=130.87, max=134.752, sum=392.611 (3)

min=0.015, mean=0.017, max=0.02, sum=0.052 (3)

min=108.578, mean=114.619, max=122.806, sum=343.856 (3)

min=0.033, mean=0.036, max=0.039, sum=0.108 (3)

min=109.32, mean=126.028, max=142.005, sum=378.084 (3)

min=0.127, mean=0.138, max=0.152, sum=0.414 (3)

min=103.173, mean=111.456, max=117.731, sum=334.367 (3)

min=0.033, mean=0.036, max=0.041, sum=0.108 (3)

min=3.056, mean=3.353, max=3.627, sum=10.059 (3)

min=346.32, mean=367.283, max=387.05, sum=1101.85 (3)

min=0.047, mean=0.053, max=0.058, sum=0.158 (3)

min=3.078, mean=3.566, max=3.9, sum=10.699 (3)

min=792.582, mean=889.249, max=939.582, sum=2667.746 (3)

min=376.13, mean=386.245, max=394.792, sum=1158.736 (3)

min=0.021, mean=0.023, max=0.026, sum=0.07 (3)

min=8.547, mean=8.635, max=8.787, sum=25.905 (3)

min=1.799, mean=1.905, max=1.982, sum=5.716 (3)

min=448.394, mean=460.351, max=469.634, sum=1381.054 (3)

min=0.019, mean=0.024, max=0.028, sum=0.071 (3)

min=4.16, mean=4.203, max=4.285, sum=12.609 (3)

min=1.749, mean=1.88, max=1.973, sum=5.64 (3)

min=451.008, mean=460.805, max=467.034, sum=1382.416 (3)

min=0.034, mean=0.04, max=0.046, sum=0.121 (3)

min=7.432, mean=7.663, max=7.833, sum=22.988 (3)

min=385.776, mean=391.689, max=396.43, sum=1175.066 (3)

min=0.014, mean=0.018, max=0.021, sum=0.054 (3)

min=1.919, mean=5.53, max=9.903, sum=16.589 (3)

min=0.136, mean=0.146, max=0.155, sum=0.439 (3)

min=116.585, mean=125.933, max=135.869, sum=377.798 (3)

min=0.016, mean=0.018, max=0.019, sum=0.054 (3)

min=161.102, mean=170.803, max=179.88, sum=512.409 (3)

min=0.07, mean=0.09, max=0.101, sum=0.27 (3)

min=3.786, mean=4.044, max=4.289, sum=12.131 (3)

min=120.087, mean=128.738, max=136.896, sum=386.215 (3)

min=0.014, mean=0.016, max=0.019, sum=0.047 (3)

min=1.212, mean=1.329, max=1.513, sum=3.987 (3)

min=160.04, mean=176.395, max=202.48, sum=529.184 (3)

min=0.006, mean=0.007, max=0.008, sum=0.02 (3)

min=1.381, mean=1.626, max=1.792, sum=4.879 (3)

min=211.16, mean=251.824, max=279.256, sum=755.472 (3)

min=0.003, mean=0.006, max=0.009, sum=0.019 (3)

min=1.185, mean=1.556, max=1.838, sum=4.668 (3)

min=174.904, mean=236.808, max=283.824, sum=710.424 (3)

min=0.464, mean=0.506, max=0.533, sum=1.518 (3)

min=91.628, mean=93.018, max=93.717, sum=279.054 (3)

min=0.4, mean=0.415, max=0.433, sum=1.246 (3)

min=2.529, mean=2.641, max=2.78, sum=7.924 (3)

min=81.888, mean=85.812, max=90.543, sum=257.436 (3)

min=0.002, mean=0.006, max=0.01, sum=0.018 (3)

min=0.402, mean=0.432, max=0.455, sum=1.295 (3)

min=37.664, mean=41.515, max=44.507, sum=124.544 (3)

min=0, mean=0.0, max=0.001, sum=0.001 (3)

min=0.33, mean=0.376, max=0.41, sum=1.128 (3)

min=27.872, mean=34.591, max=39.768, sum=103.773 (3)

min=0.003, mean=0.004, max=0.006, sum=0.012 (3)

min=0.4, mean=0.435, max=0.471, sum=1.305 (3)

min=52.211, mean=57.682, max=63.435, sum=173.047 (3)

min=0.557, mean=0.568, max=0.578, sum=1.703 (3)

min=3.012, mean=3.033, max=3.062, sum=9.1 (3)

min=111.9, mean=113.059, max=114.028, sum=339.177 (3)

min=0.039, mean=0.049, max=0.057, sum=0.147 (3)

min=0.979, mean=1.188, max=1.318, sum=3.565 (3)

min=110.496, mean=136.778, max=153.32, sum=410.335 (3)

min=0.499, mean=0.515, max=0.531, sum=1.546 (3)

min=832.035, mean=913.035, max=959.035, sum=2739.105 (3)

min=105.61, mean=108.826, max=113.005, sum=326.479 (3)

min=0.01, mean=0.01, max=0.01, sum=0.01 (1)

min=1000, mean=1000, max=1000, sum=1000 (1)

min=939.582, mean=939.582, max=939.582, sum=939.582 (1)

min=0.06, mean=0.061, max=0.063, sum=0.183 (3)

min=6.408, mean=6.761, max=7.252, sum=20.284 (3)

min=975.292, mean=1066.292, max=1167.292, sum=3198.876 (3)

min=131.212, mean=151.917, max=167.137, sum=455.75 (3)

min=0.048, mean=0.063, max=0.086, sum=0.19 (3)

min=72.708, mean=76.416, max=82.554, sum=229.248 (3)

min=0.625, mean=0.633, max=0.64, sum=1.899 (3)

min=89.512, mean=90.399, max=91.003, sum=271.196 (3)

min=4.183, mean=4.26, max=4.312, sum=12.779 (3)

min=739.652, mean=829.652, max=888.652, sum=2488.956 (3)