Evaluating equivariance to semantics-altering perturbations

robustness_contrast_sets

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# eval: Number of evaluation instances.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# train: Number of training instances (e.g., in-context examples).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# prompt tokens: Number of tokens in the prompt.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# output tokens: Actual number of output tokens.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

imdb_only_contrast:True

min=0.933, mean=0.938, max=0.949, sum=2.815 (3)

min=0.054, mean=0.067, max=0.08, sum=0.201 (3)

min=0.809, mean=0.844, max=0.874, sum=2.531 (3)

min=0.622, mean=0.68, max=0.761, sum=2.039 (3)

min=713.77, mean=859.436, max=1073.77, sum=2578.309 (3)

min=0.941, mean=0.945, max=0.949, sum=2.834 (3)

min=0.153, mean=0.17, max=0.2, sum=0.511 (3)

min=0.868, mean=0.882, max=0.902, sum=2.646 (3)

min=0.49, mean=0.526, max=0.573, sum=1.579 (3)

min=0.947, mean=0.949, max=0.949, sum=2.846 (3)

min=0.13, mean=0.149, max=0.179, sum=0.446 (3)

min=0.854, mean=0.874, max=0.885, sum=2.621 (3)

min=0.543, mean=0.591, max=0.662, sum=1.773 (3)

min=0.952, mean=0.955, max=0.958, sum=2.865 (3)

min=0.087, mean=0.112, max=0.155, sum=0.336 (3)

min=0.843, mean=0.854, max=0.865, sum=2.562 (3)

min=0.899, mean=0.904, max=0.91, sum=2.711 (3)

min=0.148, mean=0.172, max=0.187, sum=0.516 (3)

min=0.809, mean=0.817, max=0.823, sum=2.452 (3)

min=783.77, mean=1218.436, max=1675.77, sum=3655.309 (3)

min=0.935, mean=0.94, max=0.947, sum=2.82 (3)

min=0.078, mean=0.104, max=0.14, sum=0.311 (3)

min=0.795, mean=0.824, max=0.865, sum=2.472 (3)

min=0.941, mean=0.943, max=0.947, sum=2.829 (3)

min=0.141, mean=0.168, max=0.19, sum=0.504 (3)

min=0.86, mean=0.862, max=0.865, sum=2.587 (3)

min=0.93, mean=0.934, max=0.941, sum=2.801 (3)

min=0.156, mean=0.209, max=0.243, sum=0.628 (3)

min=0.829, mean=0.844, max=0.854, sum=2.531 (3)

min=1187.669, mean=1498.335, max=1747.669, sum=4495.006 (3)

min=0.941, mean=0.947, max=0.952, sum=2.84 (3)

min=0.158, mean=0.181, max=0.212, sum=0.544 (3)

min=0.865, mean=0.867, max=0.871, sum=2.601 (3)

min=0.938, mean=0.941, max=0.944, sum=2.823 (3)

min=0.105, mean=0.144, max=0.188, sum=0.431 (3)

min=0.848, mean=0.854, max=0.857, sum=2.562 (3)

min=0.927, mean=0.935, max=0.949, sum=2.806 (3)

min=0.015, mean=0.023, max=0.032, sum=0.069 (3)

min=0.817, mean=0.828, max=0.837, sum=2.483 (3)

min=0.683, mean=0.764, max=0.874, sum=2.293 (3)

min=1061.413, mean=1328.746, max=1742.413, sum=3986.239 (3)

min=1, mean=1.013, max=1.022, sum=3.039 (3)

min=0.935, mean=0.938, max=0.941, sum=2.815 (3)

min=0.283, mean=0.327, max=0.398, sum=0.981 (3)

min=0.876, mean=0.883, max=0.893, sum=2.649 (3)

min=0.921, mean=1.316, max=2.086, sum=3.949 (3)

min=1037.208, mean=1307.541, max=1706.208, sum=3922.624 (3)

T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.222, mean=0.237, max=0.264, sum=0.711 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.187, mean=0.266, max=0.346, sum=0.797 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.079, max=0.129, sum=0.236 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.343, mean=0.358, max=0.377, sum=1.073 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=356, mean=356, max=356, sum=1068 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=2.376, mean=2.877, max=3.567, sum=8.632 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0, max=0, sum=0 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=905.421, mean=911.432, max=914.486, sum=2734.295 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=5, mean=5, max=5, sum=15 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=3, mean=3, max=3, sum=9 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.935, mean=0.943, max=0.952, sum=2.829 (3)

min=0.068, mean=0.075, max=0.087, sum=0.225 (3)

min=0.86, mean=0.874, max=0.885, sum=2.621 (3)

min=0.643, mean=0.726, max=0.852, sum=2.178 (3)

min=1070.107, mean=1336.773, max=1743.107, sum=4010.32 (3)

min=0.91, mean=0.92, max=0.933, sum=2.761 (3)

min=0.075, mean=0.115, max=0.167, sum=0.344 (3)

min=0.801, mean=0.823, max=0.848, sum=2.469 (3)

min=0.456, mean=0.521, max=0.619, sum=1.563 (3)

min=0.868, mean=0.883, max=0.89, sum=2.649 (3)

min=0.107, mean=0.124, max=0.145, sum=0.371 (3)

min=0.649, mean=0.687, max=0.722, sum=2.062 (3)

min=0.372, mean=0.414, max=0.482, sum=1.243 (3)

min=0.511, mean=0.55, max=0.573, sum=1.649 (3)

min=0.09, mean=0.114, max=0.145, sum=0.341 (3)

min=0.073, mean=0.149, max=0.216, sum=0.447 (3)

min=0.396, mean=0.446, max=0.519, sum=1.339 (3)

min=0.941, mean=0.945, max=0.947, sum=2.834 (3)

min=0.025, mean=0.031, max=0.037, sum=0.092 (3)

min=0.837, mean=0.861, max=0.882, sum=2.584 (3)

min=0.949, mean=0.95, max=0.952, sum=2.851 (3)

min=0.28, mean=0.292, max=0.312, sum=0.875 (3)

min=0.871, mean=0.874, max=0.876, sum=2.621 (3)

min=1, mean=1.007, max=1.014, sum=3.02 (3)

min=0.958, mean=0.961, max=0.963, sum=2.882 (3)

min=0.015, mean=0.019, max=0.022, sum=0.057 (3)

min=0.896, mean=0.901, max=0.907, sum=2.702 (3)

min=1187.146, mean=1503.479, max=1759.146, sum=4510.438 (3)

min=0.944, mean=0.949, max=0.952, sum=2.846 (3)

min=0.026, mean=0.027, max=0.03, sum=0.082 (3)

min=0.86, mean=0.873, max=0.879, sum=2.618 (3)

min=0.944, mean=0.951, max=0.961, sum=2.854 (3)

min=0.275, mean=0.287, max=0.303, sum=0.86 (3)

min=0.885, mean=0.896, max=0.91, sum=2.688 (3)

min=0.692, mean=0.704, max=0.717, sum=2.111 (3)

min=0.933, mean=0.938, max=0.941, sum=2.815 (3)

min=0.167, mean=0.204, max=0.237, sum=0.613 (3)

min=0.398, mean=0.787, max=1.553, sum=2.361 (3)

min=1070.298, mean=1336.964, max=1749.298, sum=4010.893 (3)

min=0.346, mean=0.502, max=0.719, sum=1.506 (3)

min=0.166, mean=0.327, max=0.486, sum=0.98 (3)

min=0.202, mean=0.394, max=0.629, sum=1.183 (3)

min=0.272, mean=0.275, max=0.277, sum=0.826 (3)

min=0.472, mean=0.626, max=0.916, sum=1.879 (3)

min=0.003, mean=0.003, max=0.003, sum=0.008 (3)

min=375.444, mean=390.64, max=419.438, sum=1171.919 (3)

min=0.228, mean=0.462, max=0.699, sum=1.385 (3)

min=0.119, mean=0.323, max=0.524, sum=0.969 (3)

min=0.076, mean=0.349, max=0.593, sum=1.048 (3)

min=0.276, mean=0.281, max=0.29, sum=0.842 (3)

min=0.458, mean=0.61, max=0.89, sum=1.831 (3)

min=0.006, mean=0.006, max=0.006, sum=0.017 (3)

min=375.073, mean=390.558, max=418.14, sum=1171.674 (3)

min=0.935, mean=0.945, max=0.952, sum=2.834 (3)

min=0.15, mean=0.178, max=0.204, sum=0.535 (3)

min=0.857, mean=0.868, max=0.876, sum=2.604 (3)

min=0.626, mean=0.808, max=1.169, sum=2.424 (3)

min=0.907, mean=0.917, max=0.921, sum=2.75 (3)

min=0.278, mean=0.293, max=0.322, sum=0.878 (3)

min=0.815, mean=0.821, max=0.834, sum=2.463 (3)

min=0.254, mean=0.307, max=0.41, sum=0.922 (3)

min=0.935, mean=0.944, max=0.949, sum=2.831 (3)

min=0.059, mean=0.085, max=0.106, sum=0.254 (3)

min=0.857, mean=0.86, max=0.862, sum=2.579 (3)

min=0.938, mean=0.944, max=0.947, sum=2.831 (3)

min=0.105, mean=0.125, max=0.14, sum=0.376 (3)

min=0.857, mean=0.867, max=0.874, sum=2.601 (3)

min=0.697, mean=0.848, max=0.924, sum=2.545 (3)

min=0.057, mean=0.093, max=0.121, sum=0.278 (3)

min=0.256, mean=0.632, max=0.837, sum=1.896 (3)

min=0.214, mean=0.22, max=0.231, sum=0.66 (3)

min=0.907, mean=0.931, max=0.944, sum=2.792 (3)

min=0.125, mean=0.198, max=0.256, sum=0.595 (3)

min=0.789, mean=0.831, max=0.851, sum=2.492 (3)

min=0.102, mean=0.106, max=0.114, sum=0.319 (3)

min=0.497, mean=0.524, max=0.553, sum=1.573 (3)

min=0.25, mean=0.273, max=0.296, sum=0.819 (3)

min=0.053, mean=0.086, max=0.118, sum=0.258 (3)

min=0.123, mean=0.126, max=0.13, sum=0.377 (3)

min=0.694, mean=0.772, max=0.902, sum=2.317 (3)

min=0.141, mean=0.193, max=0.298, sum=0.58 (3)

min=0.362, mean=0.533, max=0.809, sum=1.598 (3)

min=0.141, mean=0.141, max=0.143, sum=0.424 (3)

min=0.812, mean=0.834, max=0.851, sum=2.503 (3)

min=0.121, mean=0.133, max=0.149, sum=0.399 (3)

min=0.652, mean=0.664, max=0.674, sum=1.992 (3)

min=0.944, mean=0.945, max=0.947, sum=2.834 (3)

min=0.032, mean=0.035, max=0.038, sum=0.104 (3)

min=0.885, mean=0.887, max=0.888, sum=2.66 (3)

min=0.201, mean=0.218, max=0.24, sum=0.654 (3)

min=0.919, mean=0.924, max=0.93, sum=2.772 (3)

min=0.036, mean=0.038, max=0.04, sum=0.114 (3)

min=0.806, mean=0.82, max=0.831, sum=2.461 (3)

min=0.141, mean=0.145, max=0.151, sum=0.435 (3)

min=1, mean=1.001, max=1.003, sum=3.003 (3)

min=0.907, mean=0.916, max=0.924, sum=2.747 (3)

min=0.029, mean=0.036, max=0.046, sum=0.109 (3)

min=0.784, mean=0.822, max=0.857, sum=2.466 (3)

min=0.146, mean=0.152, max=0.162, sum=0.457 (3)

min=1, mean=1.006, max=1.017, sum=3.017 (3)

min=0.823, mean=0.845, max=0.874, sum=2.534 (3)

min=0.06, mean=0.078, max=0.098, sum=0.235 (3)

min=0.601, mean=0.669, max=0.73, sum=2.006 (3)

min=0.099, mean=0.105, max=0.113, sum=0.314 (3)

min=1, mean=1.003, max=1.008, sum=3.008 (3)

min=0.893, mean=0.898, max=0.904, sum=2.694 (3)

min=0.096, mean=0.102, max=0.107, sum=0.306 (3)

min=0.725, mean=0.746, max=0.775, sum=2.239 (3)

min=1165.798, mean=1776.131, max=2443.798, sum=5328.393 (3)

min=1.017, mean=1.021, max=1.022, sum=3.062 (3)

min=0.941, mean=0.941, max=0.941, sum=0.941 (1)

min=0.258, mean=0.258, max=0.258, sum=0.258 (1)

min=0.848, mean=0.848, max=0.848, sum=0.848 (1)

min=1558.298, mean=1558.298, max=1558.298, sum=1558.298 (1)

min=0.949, mean=0.956, max=0.966, sum=2.868 (3)

min=0.115, mean=0.175, max=0.223, sum=0.525 (3)

min=0.826, mean=0.869, max=0.91, sum=2.607 (3)

min=0.864, mean=1.059, max=1.428, sum=3.178 (3)

min=1090.478, mean=1353.144, max=1760.478, sum=4059.433 (3)

min=0.93, mean=0.931, max=0.933, sum=2.792 (3)

min=0.067, mean=0.069, max=0.07, sum=0.208 (3)

min=0.857, mean=0.861, max=0.865, sum=2.584 (3)

min=1182.413, mean=1490.08, max=1736.413, sum=4470.239 (3)

min=0.994, mean=0.996, max=0.997, sum=2.989 (3)