The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

blimp

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

Exact match: Fraction of instances that the predicted output matches a correct reference exactly.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

# eval: Number of evaluation instances.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

# train: Number of training instances (e.g., in-context examples).

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

# prompt tokens: Number of tokens in the prompt.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

# output tokens: Actual number of output tokens.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design [(Warstadt et al., 2020)](https://aclanthology.org/2020.tacl-1.25/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.738, mean=0.811, max=0.927, sum=3.244 (4)

min=0.259, mean=0.26, max=0.26, sum=1.04 (4)

min=1000, mean=1000, max=1000, sum=4000 (4)

min=5.153, mean=6.764, max=7.548, sum=27.055 (4)

min=0.75, mean=0.806, max=0.903, sum=3.224 (4)

min=0.238, mean=0.239, max=0.239, sum=0.955 (4)

min=0.727, mean=0.803, max=0.911, sum=3.213 (4)

min=0.281, mean=0.283, max=0.283, sum=1.131 (4)

min=0.728, mean=0.804, max=0.927, sum=3.218 (4)

min=0.736, mean=0.81, max=0.929, sum=3.239 (4)

min=0.733, mean=0.809, max=0.935, sum=3.235 (4)

min=0.737, mean=0.816, max=0.927, sum=3.264 (4)

min=0.772, mean=0.829, max=0.937, sum=3.317 (4)

min=0.447, mean=0.452, max=0.455, sum=1.808 (4)

min=6.176, mean=8.608, max=9.793, sum=34.432 (4)

min=0.174, mean=0.383, max=0.551, sum=1.53 (4)

min=0.743, mean=0.819, max=0.929, sum=3.276 (4)

min=0.041, mean=0.448, max=1.093, sum=1.793 (4)

min=6.124, mean=8.801, max=10.03, sum=35.205 (4)

min=0.758, mean=0.83, max=0.953, sum=3.32 (4)

min=0.314, mean=0.316, max=0.316, sum=1.263 (4)

min=6.253, mean=8.843, max=10.017, sum=35.371 (4)

min=0.73, mean=0.833, max=0.962, sum=3.334 (4)

min=0.202, mean=0.202, max=0.203, sum=0.81 (4)

min=0.714, mean=0.823, max=0.953, sum=3.292 (4)

min=0.187, mean=0.188, max=0.188, sum=0.751 (4)

min=0.683, mean=0.793, max=0.927, sum=3.172 (4)

min=0.214, mean=0.214, max=0.214, sum=0.856 (4)

min=0.77, mean=0.832, max=0.934, sum=3.327 (4)

min=0.728, mean=0.832, max=0.961, sum=3.326 (4)

min=0.713, mean=0.79, max=0.904, sum=3.159 (4)

min=0.76, mean=0.805, max=0.91, sum=3.221 (4)

min=0.725, mean=0.834, max=0.938, sum=3.336 (4)

min=0.504, mean=0.541, max=0.59, sum=2.164 (4)

min=0.742, mean=0.839, max=0.951, sum=3.354 (4)

min=0.012, mean=0.48, max=1.021, sum=1.918 (4)

min=6.307, mean=8.859, max=10.1, sum=35.436 (4)

min=0.747, mean=0.831, max=0.892, sum=3.322 (4)

min=0.048, mean=0.517, max=0.986, sum=2.068 (4)

min=0.741, mean=0.827, max=0.882, sum=3.307 (4)

min=0.103, mean=0.194, max=0.318, sum=0.777 (4)

min=0.779, mean=0.826, max=0.859, sum=3.302 (4)

min=0.731, mean=0.835, max=0.96, sum=3.339 (4)

min=0.721, mean=0.84, max=0.961, sum=3.359 (4)

min=0.184, mean=0.184, max=0.184, sum=0.736 (4)

min=0.725, mean=0.832, max=0.943, sum=3.326 (4)

min=0.079, mean=0.079, max=0.08, sum=0.318 (4)

min=0.68, mean=0.833, max=0.979, sum=3.331 (4)

min=0.111, mean=0.111, max=0.111, sum=0.445 (4)

min=0.692, mean=0.818, max=0.973, sum=3.271 (4)

min=0.136, mean=0.136, max=0.136, sum=0.545 (4)

min=0.755, mean=0.823, max=0.951, sum=3.29 (4)

min=0.74, mean=0.812, max=0.89, sum=3.248 (4)

min=0.158, mean=0.158, max=0.159, sum=0.634 (4)

min=0.68, mean=0.761, max=0.812, sum=3.042 (4)

min=0.119, mean=0.12, max=0.12, sum=0.478 (4)

min=0.698, mean=0.769, max=0.823, sum=3.078 (4)

min=0.122, mean=0.122, max=0.122, sum=0.489 (4)

min=0.648, mean=0.728, max=0.774, sum=2.911 (4)

min=0.076, mean=0.076, max=0.076, sum=0.305 (4)

BLiMP (The Benchmark of Linguistic Minimal Pairs for English)

blimp_phenomenon:binding

min=0.825, mean=0.825, max=0.825, sum=0.825 (1)

min=0.26, mean=0.26, max=0.26, sum=0.26 (1)

min=1000, mean=1000, max=1000, sum=1000 (1)

min=7.512, mean=7.512, max=7.512, sum=7.512 (1)

min=0.816, mean=0.816, max=0.816, sum=0.816 (1)

min=0.239, mean=0.239, max=0.239, sum=0.239 (1)

min=0.283, mean=0.283, max=0.283, sum=0.283 (1)

min=0.83, mean=0.83, max=0.83, sum=0.83 (1)

min=0.822, mean=0.822, max=0.822, sum=0.822 (1)

min=0.826, mean=0.826, max=0.826, sum=0.826 (1)

min=0.839, mean=0.839, max=0.839, sum=0.839 (1)

min=0.453, mean=0.453, max=0.453, sum=0.453 (1)

min=9.793, mean=9.793, max=9.793, sum=9.793 (1)

min=0.416, mean=0.416, max=0.416, sum=0.416 (1)

min=0.785, mean=0.785, max=0.785, sum=0.785 (1)

min=0.043, mean=0.043, max=0.043, sum=0.043 (1)

min=9.655, mean=9.655, max=9.655, sum=9.655 (1)

min=0.801, mean=0.801, max=0.801, sum=0.801 (1)

min=0.316, mean=0.316, max=0.316, sum=0.316 (1)

min=10.017, mean=10.017, max=10.017, sum=10.017 (1)

min=0.203, mean=0.203, max=0.203, sum=0.203 (1)

min=0.805, mean=0.805, max=0.805, sum=0.805 (1)

min=0.188, mean=0.188, max=0.188, sum=0.188 (1)

min=0.765, mean=0.765, max=0.765, sum=0.765 (1)

min=0.214, mean=0.214, max=0.214, sum=0.214 (1)

min=0.824, mean=0.824, max=0.824, sum=0.824 (1)

min=0.803, mean=0.803, max=0.803, sum=0.803 (1)

min=0.778, mean=0.778, max=0.778, sum=0.778 (1)

min=0.823, mean=0.823, max=0.823, sum=0.823 (1)

min=0.59, mean=0.59, max=0.59, sum=0.59 (1)

min=0.802, mean=0.802, max=0.802, sum=0.802 (1)

min=1.021, mean=1.021, max=1.021, sum=1.021 (1)

min=10.1, mean=10.1, max=10.1, sum=10.1 (1)

min=0.797, mean=0.797, max=0.797, sum=0.797 (1)

min=0.604, mean=0.604, max=0.604, sum=0.604 (1)

min=0.318, mean=0.318, max=0.318, sum=0.318 (1)

min=0.818, mean=0.818, max=0.818, sum=0.818 (1)

min=0.813, mean=0.813, max=0.813, sum=0.813 (1)

min=0.832, mean=0.832, max=0.832, sum=0.832 (1)

min=0.184, mean=0.184, max=0.184, sum=0.184 (1)

min=0.81, mean=0.81, max=0.81, sum=0.81 (1)

min=0.08, mean=0.08, max=0.08, sum=0.08 (1)

min=0.799, mean=0.799, max=0.799, sum=0.799 (1)

min=0.111, mean=0.111, max=0.111, sum=0.111 (1)

min=0.136, mean=0.136, max=0.136, sum=0.136 (1)

min=0.159, mean=0.159, max=0.159, sum=0.159 (1)

min=0.812, mean=0.812, max=0.812, sum=0.812 (1)

min=0.12, mean=0.12, max=0.12, sum=0.12 (1)

min=0.122, mean=0.122, max=0.122, sum=0.122 (1)

min=0.726, mean=0.726, max=0.726, sum=0.726 (1)

min=0.076, mean=0.076, max=0.076, sum=0.076 (1)

phenomenon: binding

blimp_phenomenon:irregular_forms

min=0.927, mean=0.927, max=0.927, sum=0.927 (1)

min=0.259, mean=0.259, max=0.259, sum=0.259 (1)

min=5.153, mean=5.153, max=5.153, sum=5.153 (1)

min=0.903, mean=0.903, max=0.903, sum=0.903 (1)

min=0.238, mean=0.238, max=0.238, sum=0.238 (1)

min=0.911, mean=0.911, max=0.911, sum=0.911 (1)

min=0.281, mean=0.281, max=0.281, sum=0.281 (1)

min=0.929, mean=0.929, max=0.929, sum=0.929 (1)

min=0.935, mean=0.935, max=0.935, sum=0.935 (1)

min=0.937, mean=0.937, max=0.937, sum=0.937 (1)

min=0.447, mean=0.447, max=0.447, sum=0.447 (1)

min=6.176, mean=6.176, max=6.176, sum=6.176 (1)

min=0.174, mean=0.174, max=0.174, sum=0.174 (1)

min=0.041, mean=0.041, max=0.041, sum=0.041 (1)

min=6.124, mean=6.124, max=6.124, sum=6.124 (1)

min=0.953, mean=0.953, max=0.953, sum=0.953 (1)

min=0.314, mean=0.314, max=0.314, sum=0.314 (1)

min=6.253, mean=6.253, max=6.253, sum=6.253 (1)

min=0.962, mean=0.962, max=0.962, sum=0.962 (1)

min=0.202, mean=0.202, max=0.202, sum=0.202 (1)

min=0.187, mean=0.187, max=0.187, sum=0.187 (1)

min=0.934, mean=0.934, max=0.934, sum=0.934 (1)

min=0.961, mean=0.961, max=0.961, sum=0.961 (1)

min=0.904, mean=0.904, max=0.904, sum=0.904 (1)

min=0.91, mean=0.91, max=0.91, sum=0.91 (1)

min=0.938, mean=0.938, max=0.938, sum=0.938 (1)

min=0.504, mean=0.504, max=0.504, sum=0.504 (1)

min=0.951, mean=0.951, max=0.951, sum=0.951 (1)

min=0.012, mean=0.012, max=0.012, sum=0.012 (1)

min=6.307, mean=6.307, max=6.307, sum=6.307 (1)

min=0.886, mean=0.886, max=0.886, sum=0.886 (1)

min=0.048, mean=0.048, max=0.048, sum=0.048 (1)

min=0.862, mean=0.862, max=0.862, sum=0.862 (1)

min=0.11, mean=0.11, max=0.11, sum=0.11 (1)

min=0.846, mean=0.846, max=0.846, sum=0.846 (1)

min=0.96, mean=0.96, max=0.96, sum=0.96 (1)

min=0.943, mean=0.943, max=0.943, sum=0.943 (1)

min=0.079, mean=0.079, max=0.079, sum=0.079 (1)

min=0.979, mean=0.979, max=0.979, sum=0.979 (1)

min=0.973, mean=0.973, max=0.973, sum=0.973 (1)

min=0.89, mean=0.89, max=0.89, sum=0.89 (1)

min=0.158, mean=0.158, max=0.158, sum=0.158 (1)

min=0.784, mean=0.784, max=0.784, sum=0.784 (1)

min=0.119, mean=0.119, max=0.119, sum=0.119 (1)

min=0.783, mean=0.783, max=0.783, sum=0.783 (1)

min=0.774, mean=0.774, max=0.774, sum=0.774 (1)

phenomenon: irregular_forms

blimp_phenomenon:island_effects

min=0.754, mean=0.754, max=0.754, sum=0.754 (1)

min=7.548, mean=7.548, max=7.548, sum=7.548 (1)

min=0.75, mean=0.75, max=0.75, sum=0.75 (1)

min=0.733, mean=0.733, max=0.733, sum=0.733 (1)

min=0.752, mean=0.752, max=0.752, sum=0.752 (1)

min=0.741, mean=0.741, max=0.741, sum=0.741 (1)

min=0.737, mean=0.737, max=0.737, sum=0.737 (1)

min=0.772, mean=0.772, max=0.772, sum=0.772 (1)

min=0.455, mean=0.455, max=0.455, sum=0.455 (1)

min=9.532, mean=9.532, max=9.532, sum=9.532 (1)

min=0.551, mean=0.551, max=0.551, sum=0.551 (1)

min=0.743, mean=0.743, max=0.743, sum=0.743 (1)

min=0.615, mean=0.615, max=0.615, sum=0.615 (1)

min=10.03, mean=10.03, max=10.03, sum=10.03 (1)

min=0.758, mean=0.758, max=0.758, sum=0.758 (1)

min=9.879, mean=9.879, max=9.879, sum=9.879 (1)

min=0.73, mean=0.73, max=0.73, sum=0.73 (1)

min=0.714, mean=0.714, max=0.714, sum=0.714 (1)

min=0.683, mean=0.683, max=0.683, sum=0.683 (1)

min=0.77, mean=0.77, max=0.77, sum=0.77 (1)

min=0.728, mean=0.728, max=0.728, sum=0.728 (1)

min=0.713, mean=0.713, max=0.713, sum=0.713 (1)

min=0.76, mean=0.76, max=0.76, sum=0.76 (1)

min=0.725, mean=0.725, max=0.725, sum=0.725 (1)

min=0.516, mean=0.516, max=0.516, sum=0.516 (1)

min=0.742, mean=0.742, max=0.742, sum=0.742 (1)

min=0.873, mean=0.873, max=0.873, sum=0.873 (1)

min=9.864, mean=9.864, max=9.864, sum=9.864 (1)