The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

babi_qa

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

# eval: Number of evaluation instances.

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

# train: Number of training instances (e.g., in-context examples).

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

# prompt tokens: Number of tokens in the prompt.

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

# output tokens: Actual number of output tokens.

The bAbI benchmark for measuring understanding and reasoning [(Weston et al., 2015)](https://arxiv.org/pdf/1502.05698.pdf).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.506, mean=0.543, max=0.568, sum=1.63 (3)

min=0.47, mean=0.483, max=0.491, sum=1.449 (3)

min=315.789, mean=341.123, max=356.789, sum=1023.368 (3)

min=2.143, mean=2.151, max=2.166, sum=6.454 (3)

min=0.439, mean=0.469, max=0.485, sum=1.407 (3)

min=0.368, mean=0.377, max=0.382, sum=1.131 (3)

min=2.105, mean=2.113, max=2.119, sum=6.339 (3)

min=0.43, mean=0.458, max=0.5, sum=1.373 (3)

min=0.407, mean=0.415, max=0.42, sum=1.245 (3)

min=2.092, mean=2.104, max=2.116, sum=6.313 (3)

min=0.428, mean=0.47, max=0.511, sum=1.409 (3)

min=2.089, mean=2.108, max=2.131, sum=6.325 (3)

min=0.493, mean=0.521, max=0.536, sum=1.564 (3)

min=2.19, mean=2.205, max=2.226, sum=6.615 (3)

min=0.44, mean=0.455, max=0.479, sum=1.364 (3)

min=2.105, mean=2.117, max=2.127, sum=6.351 (3)

min=0.481, mean=0.504, max=0.528, sum=1.512 (3)

min=2.135, mean=2.141, max=2.145, sum=6.424 (3)

min=0.426, mean=0.452, max=0.487, sum=1.355 (3)

min=434.079, mean=470.746, max=502.079, sum=1412.238 (3)

min=1.189, mean=1.212, max=1.253, sum=3.635 (3)

min=0.446, mean=0.475, max=0.491, sum=1.424 (3)

min=1.155, mean=1.207, max=1.237, sum=3.621 (3)

min=0.471, mean=0.504, max=0.522, sum=1.511 (3)

min=1.179, mean=1.218, max=1.244, sum=3.653 (3)

min=0.425, mean=0.461, max=0.508, sum=1.382 (3)

min=0.605, mean=0.612, max=0.616, sum=1.836 (3)

min=434.355, mean=469.022, max=502.355, sum=1407.066 (3)

min=1.095, mean=1.113, max=1.127, sum=3.339 (3)

min=0.409, mean=0.447, max=0.475, sum=1.34 (3)

min=0.466, mean=0.468, max=0.469, sum=1.404 (3)

min=382.822, mean=414.156, max=443.822, sum=1242.467 (3)

min=0.165, mean=0.166, max=0.168, sum=0.498 (3)

min=4.986, mean=4.986, max=4.988, sum=14.959 (3)

min=397.457, mean=428.442, max=461.918, sum=1285.326 (3)

min=0.472, mean=0.486, max=0.498, sum=1.458 (3)

min=0.498, mean=0.51, max=0.518, sum=1.531 (3)

min=434.483, mean=469.15, max=502.483, sum=1407.45 (3)

min=1.136, mean=1.173, max=1.227, sum=3.519 (3)

min=0.322, mean=0.36, max=0.38, sum=1.081 (3)

min=0.315, mean=0.324, max=0.331, sum=0.973 (3)

min=1.344, mean=1.385, max=1.459, sum=4.156 (3)

min=0.383, mean=0.391, max=0.395, sum=1.172 (3)

min=0.279, mean=0.286, max=0.291, sum=0.858 (3)

min=1.193, mean=1.254, max=1.344, sum=3.763 (3)

min=0.291, mean=0.306, max=0.332, sum=0.918 (3)

min=0.277, mean=0.282, max=0.287, sum=0.846 (3)

min=1.363, mean=1.384, max=1.397, sum=4.151 (3)

min=0.422, mean=0.439, max=0.464, sum=1.318 (3)

min=1.107, mean=1.118, max=1.135, sum=3.353 (3)

min=0.429, mean=0.471, max=0.498, sum=1.413 (3)

min=1.209, mean=1.234, max=1.256, sum=3.702 (3)

min=0.428, mean=0.473, max=0.496, sum=1.418 (3)

min=1.243, mean=1.336, max=1.504, sum=4.007 (3)

min=0.487, mean=0.497, max=0.504, sum=1.491 (3)

min=1.149, mean=1.155, max=1.159, sum=3.464 (3)

min=0.394, mean=0.459, max=0.514, sum=1.377 (3)

min=0.195, mean=0.195, max=0.195, sum=0.585 (3)

min=0.385, mean=0.468, max=0.52, sum=1.403 (3)

min=0.326, mean=0.407, max=0.569, sum=1.222 (3)

min=434.671, mean=469.338, max=502.671, sum=1408.013 (3)

min=0.333, mean=0.412, max=0.456, sum=1.237 (3)

min=0.229, mean=0.23, max=0.231, sum=0.689 (3)

min=4.858, mean=4.878, max=4.901, sum=14.635 (3)

min=0.007, mean=0.007, max=0.007, sum=0.02 (3)

min=390.583, mean=419.224, max=449.949, sum=1257.671 (3)

min=0.485, mean=0.501, max=0.51, sum=1.502 (3)

min=0.215, mean=0.216, max=0.216, sum=0.647 (3)

min=4.851, mean=4.873, max=4.897, sum=14.619 (3)

min=394.374, mean=422.766, max=453.353, sum=1268.298 (3)

min=0.474, mean=0.507, max=0.534, sum=1.52 (3)

min=0.446, mean=0.584, max=0.858, sum=1.752 (3)

min=0.386, mean=0.408, max=0.434, sum=1.224 (3)

min=0.104, mean=0.105, max=0.105, sum=0.315 (3)

min=0.446, mean=0.481, max=0.503, sum=1.444 (3)

min=1.081, mean=1.108, max=1.129, sum=3.323 (3)

min=0.408, mean=0.411, max=0.415, sum=1.232 (3)

min=1.243, mean=1.262, max=1.273, sum=3.785 (3)

min=0.457, mean=0.462, max=0.471, sum=1.386 (3)

min=0.222, mean=0.224, max=0.227, sum=0.673 (3)

min=1.1, mean=1.134, max=1.192, sum=3.401 (3)

min=0.403, mean=0.415, max=0.43, sum=1.244 (3)

min=0.094, mean=0.095, max=0.095, sum=0.284 (3)

min=1.215, mean=1.233, max=1.254, sum=3.7 (3)

min=0.328, mean=0.348, max=0.368, sum=1.043 (3)

min=0.122, mean=0.122, max=0.123, sum=0.367 (3)

min=1.31, mean=1.349, max=1.423, sum=4.047 (3)

min=0.284, mean=0.306, max=0.34, sum=0.918 (3)

min=0.143, mean=0.144, max=0.145, sum=0.432 (3)

min=1.415, mean=1.48, max=1.593, sum=4.439 (3)

min=0.651, mean=0.653, max=0.658, sum=1.96 (3)

min=1.509, mean=1.521, max=1.533, sum=4.564 (3)

min=0.603, mean=0.618, max=0.627, sum=1.854 (3)

min=0.219, mean=0.221, max=0.223, sum=0.663 (3)

min=1.292, mean=1.311, max=1.325, sum=3.932 (3)

min=0.3, mean=0.384, max=0.443, sum=1.153 (3)

min=0.133, mean=0.135, max=0.136, sum=0.404 (3)

min=1.125, mean=1.286, max=1.538, sum=3.857 (3)

min=0.279, mean=0.289, max=0.305, sum=0.866 (3)

min=0.136, mean=0.137, max=0.138, sum=0.411 (3)

min=1.554, mean=1.605, max=1.691, sum=4.816 (3)

min=0.187, mean=0.226, max=0.252, sum=0.679 (3)

min=0.089, mean=0.091, max=0.094, sum=0.274 (3)

min=1.206, mean=1.435, max=1.883, sum=4.304 (3)

min=0.677, mean=0.686, max=0.7, sum=2.058 (3)

min=0.189, mean=0.194, max=0.199, sum=0.582 (3)

min=2.062, mean=2.074, max=2.084, sum=6.221 (3)

min=0.462, mean=0.481, max=0.491, sum=1.443 (3)

min=0.108, mean=0.109, max=0.11, sum=0.328 (3)

min=2.13, mean=2.131, max=2.132, sum=6.394 (3)

min=0.435, mean=0.492, max=0.525, sum=1.476 (3)

min=382.233, mean=412.9, max=444.233, sum=1238.699 (3)

min=2.746, mean=2.886, max=3.111, sum=8.659 (3)

min=0.466, mean=0.466, max=0.466, sum=0.466 (1)

min=434.671, mean=434.671, max=434.671, sum=434.671 (1)

min=0.422, mean=0.443, max=0.459, sum=1.33 (3)

min=0.479, mean=0.491, max=0.514, sum=1.473 (3)

min=375.649, mean=405.316, max=439.649, sum=1215.947 (3)

min=2.14, mean=2.161, max=2.177, sum=6.482 (3)

min=0.459, mean=0.468, max=0.474, sum=1.404 (3)

min=1.128, mean=1.191, max=1.249, sum=3.573 (3)

min=0.7, mean=0.702, max=0.704, sum=2.107 (3)

min=1.469, mean=1.479, max=1.488, sum=4.438 (3)

min=0.339, mean=0.346, max=0.352, sum=1.038 (3)

min=0.283, mean=0.283, max=0.284, sum=0.849 (3)

min=434.321, mean=468.988, max=502.321, sum=1406.964 (3)

bAbI

babi_qa_task:15

min=0.411, mean=0.462, max=0.498, sum=1.385 (3)

min=0.472, mean=0.473, max=0.474, sum=1.418 (3)

min=330.744, mean=332.077, max=333.744, sum=996.231 (3)

min=0.438, mean=0.467, max=0.504, sum=1.402 (3)

min=0.372, mean=0.372, max=0.373, sum=1.117 (3)

min=0.322, mean=0.364, max=0.409, sum=1.092 (3)

min=0.409, mean=0.41, max=0.41, sum=1.229 (3)

min=0.539, mean=0.561, max=0.576, sum=1.682 (3)

min=0.345, mean=0.387, max=0.446, sum=1.16 (3)

min=0.422, mean=0.461, max=0.488, sum=1.384 (3)

min=0.262, mean=0.285, max=0.305, sum=0.856 (3)

min=0.249, mean=0.254, max=0.263, sum=0.761 (3)

min=455.964, mean=456.964, max=457.964, sum=1370.891 (3)

min=0.321, mean=0.356, max=0.377, sum=1.069 (3)

min=0.278, mean=0.31, max=0.328, sum=0.93 (3)

min=0.483, mean=0.565, max=0.647, sum=1.694 (3)

min=0.617, mean=0.618, max=0.619, sum=1.853 (3)

min=464.033, mean=465.7, max=469.033, sum=1397.099 (3)

min=0.332, mean=0.339, max=0.343, sum=1.017 (3)

min=0.188, mean=0.599, max=1.399, sum=1.796 (3)

min=400.628, mean=402.295, max=404.628, sum=1206.884 (3)

min=0.358, mean=0.366, max=0.377, sum=1.099 (3)

min=463.824, mean=464.157, max=464.824, sum=1392.472 (3)

min=0.277, mean=0.324, max=0.37, sum=0.972 (3)

min=0.51, mean=0.51, max=0.512, sum=1.531 (3)

min=464.255, mean=465.922, max=469.255, sum=1397.766 (3)

min=0.216, mean=0.22, max=0.223, sum=0.659 (3)

min=0.315, mean=0.316, max=0.316, sum=0.947 (3)

min=0.198, mean=0.249, max=0.325, sum=0.746 (3)

min=0.284, mean=0.284, max=0.285, sum=0.853 (3)

min=0.215, mean=0.215, max=0.215, sum=0.644 (3)

min=0.277, mean=0.278, max=0.278, sum=0.833 (3)

min=0.332, mean=0.353, max=0.383, sum=1.058 (3)

min=0.22, mean=0.25, max=0.272, sum=0.751 (3)

min=0.391, mean=0.42, max=0.465, sum=1.26 (3)

min=0.377, mean=0.407, max=0.454, sum=1.221 (3)

min=0.427, mean=0.46, max=0.476, sum=1.38 (3)

min=0.109, mean=0.27, max=0.592, sum=0.81 (3)

min=0.422, mean=0.443, max=0.459, sum=1.328 (3)

min=0.319, mean=0.514, max=0.903, sum=1.542 (3)

min=479.45, mean=480.45, max=482.45, sum=1441.35 (3)

min=0.259, mean=0.28, max=0.294, sum=0.839 (3)

min=0.257, mean=0.261, max=0.27, sum=0.784 (3)

min=0.325, mean=0.345, max=0.361, sum=1.034 (3)

min=0.218, mean=0.218, max=0.219, sum=0.655 (3)

min=467.824, mean=468.157, max=468.824, sum=1404.472 (3)

min=0.213, mean=0.241, max=0.268, sum=0.724 (3)

min=0.365, mean=0.564, max=0.856, sum=1.692 (3)

min=0.365, mean=0.374, max=0.388, sum=1.123 (3)

min=0.155, mean=0.174, max=0.201, sum=0.522 (3)

min=0.361, mean=0.432, max=0.47, sum=1.296 (3)

min=0.321, mean=0.345, max=0.386, sum=1.035 (3)

min=0.437, mean=0.507, max=0.567, sum=1.521 (3)

min=0.224, mean=0.224, max=0.224, sum=0.672 (3)

min=0.284, mean=0.316, max=0.367, sum=0.948 (3)

min=0.094, mean=0.094, max=0.094, sum=0.281 (3)

min=0.261, mean=0.298, max=0.349, sum=0.893 (3)

min=0.121, mean=0.121, max=0.121, sum=0.363 (3)

min=0.231, mean=0.273, max=0.344, sum=0.82 (3)

min=0.141, mean=0.141, max=0.141, sum=0.424 (3)

min=0.494, mean=0.546, max=0.602, sum=1.639 (3)

min=0.429, mean=0.558, max=0.653, sum=1.674 (3)

min=0.219, mean=0.219, max=0.219, sum=0.657 (3)