In sentiment classification, given a text (e.g., movie review), the goal is to predict the sentiment (positive or negative).

sentiment_analysis

How many models this model outperform on average (over columns).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

accuracy

min=0.934, mean=0.943, max=0.951, sum=2.83 (3)

min=0.951, mean=0.956, max=0.962, sum=2.869 (3)

min=0.947, mean=0.953, max=0.957, sum=2.859 (3)

min=0.947, mean=0.957, max=0.964, sum=2.872 (3)

min=0.936, mean=0.938, max=0.943, sum=2.815 (3)

min=0.926, mean=0.938, max=0.954, sum=2.814 (3)

min=0.952, mean=0.956, max=0.96, sum=2.869 (3)

min=0.931, mean=0.939, max=0.949, sum=2.818 (3)

min=0.944, mean=0.947, max=0.951, sum=2.842 (3)

min=0.957, mean=0.959, max=0.961, sum=2.878 (3)

min=0.924, mean=0.934, max=0.948, sum=2.802 (3)

min=0.936, mean=0.945, max=0.95, sum=2.836 (3)

min=0.181, mean=0.207, max=0.26, sum=0.622 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.941, mean=0.956, max=0.965, sum=2.867 (3)

min=0.929, mean=0.933, max=0.94, sum=2.8 (3)

min=0.917, mean=0.935, max=0.947, sum=2.805 (3)

min=0.53, mean=0.578, max=0.618, sum=1.735 (3)

min=0.941, mean=0.956, max=0.965, sum=2.868 (3)

min=0.917, mean=0.935, max=0.947, sum=2.804 (3)

min=0.959, mean=0.961, max=0.962, sum=2.882 (3)

min=0.955, mean=0.96, max=0.965, sum=2.881 (3)

min=0.932, mean=0.939, max=0.946, sum=2.816 (3)

min=0.946, mean=0.948, max=0.95, sum=2.844 (3)

min=0.248, mean=0.379, max=0.568, sum=1.137 (3)

min=0.13, mean=0.337, max=0.556, sum=1.01 (3)

min=0.932, mean=0.947, max=0.96, sum=2.842 (3)

min=0.906, mean=0.917, max=0.926, sum=2.752 (3)

min=0.939, mean=0.941, max=0.942, sum=2.822 (3)

min=0.923, mean=0.927, max=0.934, sum=2.782 (3)

min=0.925, mean=0.933, max=0.942, sum=2.8 (3)

min=0.831, mean=0.889, max=0.939, sum=2.668 (3)

min=0.5, mean=0.597, max=0.646, sum=1.792 (3)

min=0.834, mean=0.849, max=0.861, sum=2.547 (3)

min=0.828, mean=0.848, max=0.881, sum=2.545 (3)

min=0.945, mean=0.948, max=0.953, sum=2.843 (3)

min=0.915, mean=0.923, max=0.927, sum=2.768 (3)

min=0.902, mean=0.913, max=0.921, sum=2.738 (3)

min=0.776, mean=0.822, max=0.853, sum=2.466 (3)

min=0.899, mean=0.907, max=0.912, sum=2.72 (3)

min=0.907, mean=0.907, max=0.907, sum=0.907 (1)

min=0.946, mean=0.955, max=0.961, sum=2.864 (3)

min=0.936, mean=0.94, max=0.946, sum=2.821 (3)

min=0.928, mean=0.935, max=0.939, sum=2.806 (3)

min=0.776, mean=0.836, max=0.876, sum=2.509 (3)

Accuracy

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

calibration

min=0.06, mean=0.064, max=0.072, sum=0.191 (3)

min=0.196, mean=0.213, max=0.234, sum=0.639 (3)

min=0.152, mean=0.158, max=0.166, sum=0.473 (3)

min=0.119, mean=0.136, max=0.165, sum=0.407 (3)

min=0.157, mean=0.182, max=0.199, sum=0.546 (3)

min=0.088, mean=0.111, max=0.153, sum=0.333 (3)

min=0.153, mean=0.178, max=0.201, sum=0.534 (3)

min=0.187, mean=0.232, max=0.257, sum=0.695 (3)

min=0.177, mean=0.204, max=0.232, sum=0.612 (3)

min=0.137, mean=0.173, max=0.222, sum=0.519 (3)

min=0.01, mean=0.015, max=0.024, sum=0.045 (3)

min=0.305, mean=0.343, max=0.41, sum=1.029 (3)

min=0.207, mean=0.291, max=0.36, sum=0.872 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.05, mean=0.069, max=0.081, sum=0.206 (3)

min=0.098, mean=0.132, max=0.183, sum=0.396 (3)

min=0.335, mean=0.36, max=0.394, sum=1.08 (3)

min=0.085, mean=0.134, max=0.174, sum=0.401 (3)

min=0.05, mean=0.069, max=0.082, sum=0.207 (3)

min=0.335, mean=0.36, max=0.394, sum=1.079 (3)

min=0.011, mean=0.014, max=0.019, sum=0.043 (3)

min=0.011, mean=0.015, max=0.02, sum=0.045 (3)

min=0.285, mean=0.295, max=0.311, sum=0.884 (3)

min=0.189, mean=0.23, max=0.269, sum=0.69 (3)

min=0.108, mean=0.236, max=0.374, sum=0.707 (3)

min=0.037, mean=0.225, max=0.41, sum=0.675 (3)

min=0.164, mean=0.19, max=0.216, sum=0.569 (3)

min=0.289, mean=0.302, max=0.327, sum=0.905 (3)

min=0.065, mean=0.087, max=0.106, sum=0.262 (3)

min=0.093, mean=0.118, max=0.136, sum=0.355 (3)

min=0.104, mean=0.126, max=0.166, sum=0.378 (3)

min=0.231, mean=0.259, max=0.285, sum=0.776 (3)

min=0.161, mean=0.212, max=0.289, sum=0.637 (3)

min=0.223, mean=0.274, max=0.332, sum=0.821 (3)

min=0.086, mean=0.113, max=0.132, sum=0.339 (3)

min=0.029, mean=0.031, max=0.033, sum=0.092 (3)

min=0.027, mean=0.031, max=0.034, sum=0.093 (3)

min=0.028, mean=0.038, max=0.05, sum=0.115 (3)

min=0.053, mean=0.09, max=0.142, sum=0.269 (3)

min=0.088, mean=0.093, max=0.101, sum=0.28 (3)

min=0.248, mean=0.248, max=0.248, sum=0.248 (1)

min=0.117, mean=0.18, max=0.225, sum=0.541 (3)

min=0.054, mean=0.06, max=0.064, sum=0.179 (3)

min=0.061, mean=0.065, max=0.072, sum=0.194 (3)

min=0.369, mean=0.418, max=0.496, sum=1.255 (3)

Calibration

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

robustness

min=0.917, mean=0.923, max=0.934, sum=2.768 (3)

min=0.927, mean=0.932, max=0.936, sum=2.796 (3)

min=0.932, mean=0.941, max=0.948, sum=2.822 (3)

min=0.931, mean=0.947, max=0.955, sum=2.841 (3)

min=0.878, mean=0.896, max=0.916, sum=2.688 (3)

min=0.915, mean=0.928, max=0.949, sum=2.784 (3)

min=0.935, mean=0.941, max=0.946, sum=2.822 (3)

min=0.864, mean=0.887, max=0.918, sum=2.662 (3)

min=0.913, mean=0.92, max=0.933, sum=2.76 (3)

min=0.931, mean=0.932, max=0.934, sum=2.797 (3)

min=0.921, mean=0.928, max=0.94, sum=2.783 (3)

min=0.907, mean=0.92, max=0.927, sum=2.761 (3)

min=0.13, mean=0.17, max=0.227, sum=0.511 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.907, mean=0.923, max=0.933, sum=2.768 (3)

min=0.895, mean=0.902, max=0.91, sum=2.706 (3)

min=0.878, mean=0.889, max=0.897, sum=2.666 (3)

min=0.447, mean=0.473, max=0.498, sum=1.418 (3)

min=0.907, mean=0.923, max=0.933, sum=2.769 (3)

min=0.878, mean=0.888, max=0.896, sum=2.665 (3)

min=0.917, mean=0.921, max=0.925, sum=2.762 (3)

min=0.929, mean=0.933, max=0.936, sum=2.799 (3)

min=0.895, mean=0.903, max=0.908, sum=2.709 (3)

min=0.906, mean=0.912, max=0.921, sum=2.736 (3)

min=0.17, mean=0.304, max=0.51, sum=0.911 (3)

min=0.091, mean=0.276, max=0.485, sum=0.827 (3)

min=0.904, mean=0.919, max=0.937, sum=2.756 (3)

min=0.872, mean=0.886, max=0.901, sum=2.659 (3)

min=0.92, mean=0.921, max=0.922, sum=2.763 (3)

min=0.883, mean=0.896, max=0.909, sum=2.687 (3)

min=0.855, mean=0.873, max=0.89, sum=2.62 (3)

min=0.716, mean=0.803, max=0.892, sum=2.41 (3)

min=0.476, mean=0.5, max=0.512, sum=1.5 (3)

min=0.663, mean=0.701, max=0.737, sum=2.102 (3)

min=0.749, mean=0.779, max=0.827, sum=2.338 (3)

min=0.898, mean=0.925, max=0.946, sum=2.776 (3)

min=0.876, mean=0.881, max=0.887, sum=2.642 (3)

min=0.821, mean=0.844, max=0.868, sum=2.532 (3)

min=0.663, mean=0.716, max=0.744, sum=2.148 (3)

min=0.857, mean=0.875, max=0.889, sum=2.626 (3)

min=0.843, mean=0.843, max=0.843, sum=0.843 (1)

min=0.921, mean=0.938, max=0.955, sum=2.814 (3)

min=0.898, mean=0.906, max=0.916, sum=2.718 (3)

min=0.898, mean=0.905, max=0.909, sum=2.715 (3)

min=0.578, mean=0.719, max=0.79, sum=2.158 (3)

Robustness

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

fairness

min=0.922, mean=0.932, max=0.941, sum=2.797 (3)

min=0.939, mean=0.946, max=0.951, sum=2.839 (3)

min=0.94, mean=0.946, max=0.95, sum=2.839 (3)

min=0.935, mean=0.95, max=0.959, sum=2.851 (3)

min=0.928, mean=0.933, max=0.937, sum=2.799 (3)

min=0.92, mean=0.931, max=0.951, sum=2.792 (3)

min=0.941, mean=0.945, max=0.951, sum=2.835 (3)

min=0.902, mean=0.912, max=0.926, sum=2.737 (3)

min=0.93, mean=0.937, max=0.946, sum=2.811 (3)

min=0.948, mean=0.949, max=0.951, sum=2.848 (3)

min=0.918, mean=0.925, max=0.936, sum=2.775 (3)

min=0.927, mean=0.938, max=0.946, sum=2.814 (3)

min=0.129, mean=0.168, max=0.22, sum=0.505 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.93, mean=0.949, max=0.96, sum=2.846 (3)

min=0.912, mean=0.92, max=0.93, sum=2.759 (3)

min=0.896, mean=0.918, max=0.936, sum=2.753 (3)

min=0.49, mean=0.518, max=0.54, sum=1.554 (3)

min=0.931, mean=0.949, max=0.96, sum=2.847 (3)

min=0.896, mean=0.917, max=0.936, sum=2.752 (3)

min=0.946, mean=0.95, max=0.954, sum=2.851 (3)

min=0.951, mean=0.957, max=0.96, sum=2.871 (3)

min=0.92, mean=0.927, max=0.932, sum=2.782 (3)

min=0.925, mean=0.928, max=0.933, sum=2.785 (3)

min=0.162, mean=0.303, max=0.502, sum=0.91 (3)

min=0.092, mean=0.271, max=0.484, sum=0.814 (3)

min=0.929, mean=0.944, max=0.958, sum=2.831 (3)

min=0.898, mean=0.908, max=0.919, sum=2.725 (3)

min=0.933, mean=0.936, max=0.94, sum=2.807 (3)

min=0.904, mean=0.912, max=0.922, sum=2.737 (3)

min=0.917, mean=0.921, max=0.923, sum=2.762 (3)

min=0.792, mean=0.86, max=0.922, sum=2.581 (3)

min=0.489, mean=0.534, max=0.558, sum=1.602 (3)

min=0.787, mean=0.806, max=0.819, sum=2.417 (3)

min=0.814, mean=0.833, max=0.868, sum=2.498 (3)

min=0.919, mean=0.934, max=0.945, sum=2.803 (3)

min=0.903, mean=0.91, max=0.916, sum=2.731 (3)

min=0.871, mean=0.887, max=0.901, sum=2.66 (3)

min=0.724, mean=0.769, max=0.808, sum=2.308 (3)

min=0.844, mean=0.863, max=0.886, sum=2.59 (3)

min=0.89, mean=0.89, max=0.89, sum=0.89 (1)

min=0.92, mean=0.933, max=0.951, sum=2.799 (3)

min=0.925, mean=0.931, max=0.94, sum=2.793 (3)

min=0.909, mean=0.918, max=0.923, sum=2.754 (3)

min=0.709, mean=0.8, max=0.853, sum=2.4 (3)

Fairness

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

bias

Bias

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

toxicity

Toxicity

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.