Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

synthetic_reasoning_natural

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

F1 (set match): Average F1 score in terms of set overlap between the model predicted set and correct reference set.

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# eval: Number of evaluation instances.

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# train: Number of training instances (e.g., in-context examples).

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# prompt tokens: Number of tokens in the prompt.

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# output tokens: Actual number of output tokens.

Synthetic reasoning tasks defined using simple natural language based on LIME [(Wu et al., 2021)](https://proceedings.mlr.press/v139/wu21c.html).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.12, mean=0.174, max=0.239, sum=1.047 (6)

min=0.569, mean=0.591, max=0.61, sum=3.544 (6)

min=312.217, mean=317.547, max=322.217, sum=1905.283 (6)

min=4.412, mean=4.987, max=5.449, sum=29.922 (6)

min=0.126, mean=0.154, max=0.183, sum=0.922 (6)

min=0.421, mean=0.424, max=0.426, sum=2.541 (6)

min=5.476, mean=5.528, max=5.559, sum=33.171 (6)

min=0.102, mean=0.154, max=0.234, sum=0.926 (6)

min=0.474, mean=0.478, max=0.481, sum=2.865 (6)

min=5.361, mean=5.47, max=5.551, sum=32.821 (6)

min=0.06, mean=0.139, max=0.204, sum=0.834 (6)

min=5.291, mean=5.442, max=5.55, sum=32.654 (6)

min=0.058, mean=0.144, max=0.245, sum=0.865 (6)

min=5.295, mean=5.461, max=5.553, sum=32.765 (6)

min=0.111, mean=0.164, max=0.231, sum=0.983 (6)

min=5.493, mean=5.534, max=5.571, sum=33.202 (6)

min=0.142, mean=0.176, max=0.199, sum=1.057 (6)

min=3.441, mean=4.462, max=5.124, sum=26.775 (6)

min=392.445, mean=413.838, max=427.445, sum=2483.025 (6)

min=0.133, mean=0.259, max=0.396, sum=1.554 (6)

min=0.806, mean=0.82, max=0.83, sum=4.921 (6)

min=383.15, mean=400.15, max=411.15, sum=2400.897 (6)

min=4.74, mean=4.989, max=5.082, sum=29.932 (6)

min=0.115, mean=0.197, max=0.3, sum=1.182 (6)

min=0.272, mean=0.292, max=0.326, sum=1.749 (6)

min=363.423, mean=378.218, max=391.014, sum=2269.311 (6)

min=9, mean=9.026, max=9.155, sum=54.155 (6)

min=0, mean=0.002, max=0.006, sum=0.011 (6)

min=0.262, mean=0.269, max=0.276, sum=1.614 (6)

min=373.103, mean=402.929, max=432.755, sum=2417.575 (6)

min=0.477, mean=0.485, max=0.492, sum=2.912 (6)

min=389.033, mean=407.57, max=424.033, sum=2445.419 (6)

min=0.296, mean=0.301, max=0.305, sum=1.807 (6)

min=0.268, mean=0.272, max=0.275, sum=1.632 (6)

min=0.267, mean=0.27, max=0.272, sum=1.617 (6)

min=0.185, mean=0.254, max=0.303, sum=1.522 (6)

min=5.06, mean=5.094, max=5.173, sum=30.561 (6)

min=0.172, mean=0.245, max=0.355, sum=1.469 (6)

min=5.05, mean=5.08, max=5.177, sum=30.482 (6)

min=0.158, mean=0.199, max=0.25, sum=1.193 (6)

min=0.09, mean=0.1, max=0.114, sum=0.598 (6)

min=9, mean=9.611, max=10.584, sum=57.666 (6)

min=0.117, mean=0.167, max=0.199, sum=1.004 (6)

min=0.095, mean=0.104, max=0.111, sum=0.626 (6)

min=383.918, mean=403.046, max=418.918, sum=2418.274 (6)

min=9, mean=9.2, max=9.423, sum=55.202 (6)

min=0.015, mean=0.101, max=0.174, sum=0.605 (6)

min=0.44, mean=0.452, max=0.466, sum=2.711 (6)

min=372.103, mean=401.929, max=431.755, sum=2411.575 (6)

min=0.148, mean=0.187, max=0.258, sum=1.119 (6)

min=0.341, mean=0.341, max=0.342, sum=2.049 (6)

min=376.103, mean=405.929, max=435.755, sum=2435.575 (6)

min=0.115, mean=0.248, max=0.408, sum=1.487 (6)

min=0.212, mean=0.218, max=0.231, sum=1.306 (6)

min=9, mean=9.228, max=10.212, sum=55.367 (6)

min=0.168, mean=0.213, max=0.288, sum=1.278 (6)

min=0.065, mean=0.067, max=0.071, sum=0.404 (6)

min=0.157, mean=0.243, max=0.329, sum=1.455 (6)

min=3.738, mean=4.536, max=4.938, sum=27.214 (6)

min=0.201, mean=0.247, max=0.355, sum=1.482 (6)

min=5.017, mean=5.046, max=5.08, sum=30.276 (6)

min=0.083, mean=0.165, max=0.243, sum=0.988 (6)

min=0.334, mean=0.339, max=0.341, sum=2.032 (6)

min=4.895, mean=5.018, max=5.066, sum=30.109 (6)

min=0.11, mean=0.149, max=0.184, sum=0.894 (6)

min=0.117, mean=0.118, max=0.119, sum=0.71 (6)

min=4.623, mean=4.839, max=5.074, sum=29.033 (6)

min=0.106, mean=0.12, max=0.133, sum=0.719 (6)

min=0.143, mean=0.146, max=0.148, sum=0.873 (6)

min=4.862, mean=5.328, max=5.722, sum=31.969 (6)

min=0.066, mean=0.088, max=0.118, sum=0.53 (6)

min=0.166, mean=0.169, max=0.172, sum=1.014 (6)

min=5.27, mean=5.725, max=6.144, sum=34.35 (6)

min=0.598, mean=0.734, max=0.831, sum=4.401 (6)

min=4.917, mean=5.01, max=5.181, sum=30.06 (6)

min=0.499, mean=0.623, max=0.732, sum=3.738 (6)

min=0.328, mean=0.332, max=0.336, sum=1.99 (6)

min=5.093, mean=5.191, max=5.383, sum=31.146 (6)

min=0.164, mean=0.221, max=0.3, sum=1.323 (6)

min=0.158, mean=0.159, max=0.159, sum=0.952 (6)

min=5.021, mean=5.115, max=5.212, sum=30.687 (6)

min=0.169, mean=0.212, max=0.261, sum=1.27 (6)

min=0.16, mean=0.161, max=0.162, sum=0.966 (6)

min=5.099, mean=5.238, max=5.369, sum=31.427 (6)

min=0.072, mean=0.145, max=0.241, sum=0.872 (6)

min=0.113, mean=0.116, max=0.121, sum=0.693 (6)

min=5.353, mean=5.771, max=6.753, sum=34.627 (6)

min=0.575, mean=0.684, max=0.79, sum=4.102 (6)

min=0.156, mean=0.158, max=0.159, sum=0.948 (6)

min=0.089, mean=0.164, max=0.238, sum=0.984 (6)

min=0.098, mean=0.099, max=0.099, sum=0.591 (6)

min=0.472, mean=0.604, max=0.73, sum=3.624 (6)

min=339.394, mean=358.219, max=372.394, sum=2149.315 (6)

min=5.54, mean=5.902, max=6.623, sum=35.414 (6)

min=383.918, mean=388.046, max=392.173, sum=776.091 (2)

min=0.221, mean=0.254, max=0.299, sum=1.521 (6)

min=0.312, mean=0.476, max=0.624, sum=2.857 (6)

min=345.326, mean=366.73, max=380.468, sum=2200.383 (6)

min=5.843, mean=5.859, max=5.87, sum=35.151 (6)

min=0.129, mean=0.247, max=0.396, sum=1.481 (6)

min=5.017, mean=5.04, max=5.068, sum=30.241 (6)

min=0.43, mean=0.576, max=0.729, sum=3.454 (6)

min=5.128, mean=5.288, max=5.449, sum=31.726 (6)

min=0.041, mean=0.061, max=0.087, sum=0.369 (6)

min=0.18, mean=0.207, max=0.222, sum=1.244 (6)

min=383.606, mean=402.441, max=416.606, sum=2414.649 (6)

min=13.353, mean=15.611, max=17.907, sum=93.668 (6)

Synthetic reasoning (natural language)

synthetic_reasoning_natural_difficulty:easy

min=0.188, mean=0.212, max=0.239, sum=0.637 (3)

min=0.59, mean=0.599, max=0.605, sum=1.796 (3)

min=312.217, mean=318.217, max=322.217, sum=954.652 (3)

min=4.911, mean=5.166, max=5.305, sum=15.497 (3)

min=0.156, mean=0.168, max=0.183, sum=0.505 (3)

min=0.421, mean=0.424, max=0.426, sum=1.271 (3)

min=5.476, mean=5.52, max=5.559, sum=16.561 (3)

min=0.136, mean=0.189, max=0.234, sum=0.567 (3)

min=0.474, mean=0.478, max=0.481, sum=1.433 (3)

min=5.39, mean=5.458, max=5.54, sum=16.373 (3)

min=0.136, mean=0.174, max=0.204, sum=0.523 (3)

min=5.291, mean=5.397, max=5.524, sum=16.192 (3)

min=0.178, mean=0.209, max=0.245, sum=0.627 (3)

min=5.295, mean=5.369, max=5.406, sum=16.107 (3)

min=0.128, mean=0.194, max=0.231, sum=0.583 (3)

min=5.493, mean=5.522, max=5.571, sum=16.567 (3)

min=0.17, mean=0.185, max=0.199, sum=0.556 (3)

min=3.441, mean=3.909, max=4.208, sum=11.726 (3)

min=400.897, mean=412.897, max=421.897, sum=1238.691 (3)

min=0.232, mean=0.321, max=0.396, sum=0.963 (3)

min=0.806, mean=0.816, max=0.827, sum=2.449 (3)

min=391.15, mean=399.15, max=406.15, sum=1197.449 (3)

min=4.74, mean=4.927, max=5.064, sum=14.781 (3)

min=0.246, mean=0.266, max=0.3, sum=0.799 (3)

min=0.281, mean=0.299, max=0.326, sum=0.897 (3)

min=363.423, mean=375.423, max=388.423, sum=1126.27 (3)

min=0, mean=0.002, max=0.006, sum=0.006 (3)

min=0.276, mean=0.276, max=0.276, sum=0.827 (3)

min=373.755, mean=410.755, max=432.755, sum=1232.266 (3)

min=0.48, mean=0.484, max=0.487, sum=1.452 (3)

min=396.107, mean=404.107, max=411.107, sum=1212.32 (3)

min=0.298, mean=0.3, max=0.302, sum=0.901 (3)

min=0.27, mean=0.271, max=0.273, sum=0.814 (3)

min=0.268, mean=0.269, max=0.27, sum=0.807 (3)

min=0.298, mean=0.3, max=0.303, sum=0.9 (3)

min=5.06, mean=5.062, max=5.066, sum=15.186 (3)

min=0.234, mean=0.289, max=0.355, sum=0.866 (3)

min=5.056, mean=5.056, max=5.056, sum=15.169 (3)

min=0.21, mean=0.227, max=0.25, sum=0.68 (3)

min=0.091, mean=0.098, max=0.102, sum=0.293 (3)

min=9, mean=9.526, max=10.025, sum=28.577 (3)

min=0.174, mean=0.187, max=0.199, sum=0.561 (3)

min=0.105, mean=0.108, max=0.109, sum=0.323 (3)

min=392.173, mean=400.173, max=407.173, sum=1200.518 (3)

min=9, mean=9.153, max=9.311, sum=27.458 (3)

min=0.015, mean=0.083, max=0.174, sum=0.248 (3)

min=0.46, mean=0.464, max=0.466, sum=1.391 (3)

min=372.755, mean=409.755, max=431.755, sum=1229.266 (3)

min=0.178, mean=0.217, max=0.258, sum=0.651 (3)

min=0.341, mean=0.342, max=0.342, sum=1.025 (3)

min=376.755, mean=413.755, max=435.755, sum=1241.266 (3)

min=0.322, mean=0.361, max=0.408, sum=1.082 (3)

min=0.212, mean=0.22, max=0.231, sum=0.661 (3)

min=9, mean=9.404, max=10.212, sum=28.212 (3)

min=0.218, mean=0.249, max=0.288, sum=0.746 (3)

min=0.065, mean=0.067, max=0.07, sum=0.201 (3)

min=9, mean=9.052, max=9.155, sum=27.155 (3)

min=0.273, mean=0.297, max=0.329, sum=0.891 (3)

min=3.738, mean=4.452, max=4.938, sum=13.357 (3)

min=0.216, mean=0.287, max=0.355, sum=0.86 (3)

min=5.017, mean=5.019, max=5.021, sum=15.056 (3)

min=0.136, mean=0.201, max=0.243, sum=0.603 (3)

min=0.334, mean=0.337, max=0.339, sum=1.012 (3)

min=4.895, mean=4.977, max=5.017, sum=14.93 (3)

min=0.11, mean=0.155, max=0.184, sum=0.465 (3)

min=0.118, mean=0.119, max=0.119, sum=0.356 (3)

min=4.757, mean=4.863, max=4.996, sum=14.59 (3)

min=0.115, mean=0.124, max=0.133, sum=0.372 (3)

min=0.144, mean=0.145, max=0.146, sum=0.436 (3)

min=5.06, mean=5.273, max=5.404, sum=15.819 (3)

min=0.092, mean=0.106, max=0.118, sum=0.319 (3)

min=0.168, mean=0.17, max=0.172, sum=0.51 (3)

min=5.534, mean=5.871, max=6.144, sum=17.612 (3)

min=0.823, mean=0.826, max=0.831, sum=2.477 (3)

min=4.917, mean=5.037, max=5.181, sum=15.111 (3)

min=0.704, mean=0.72, max=0.732, sum=2.16 (3)

min=0.328, mean=0.329, max=0.33, sum=0.987 (3)

min=5.093, mean=5.105, max=5.113, sum=15.315 (3)

min=0.219, mean=0.268, max=0.3, sum=0.803 (3)

min=0.158, mean=0.159, max=0.159, sum=0.476 (3)

min=5.021, mean=5.096, max=5.212, sum=15.289 (3)

min=0.225, mean=0.242, max=0.261, sum=0.727 (3)