The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

imdb

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# eval: Number of evaluation instances.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# train: Number of training instances (e.g., in-context examples).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# prompt tokens: Number of tokens in the prompt.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# output tokens: Actual number of output tokens.

The IMDB benchmark for sentiment analysis in movie review [(Maas et al., 2011)](https://aclanthology.org/P11-1015/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.934, mean=0.943, max=0.951, sum=2.83 (3)

min=0.06, mean=0.064, max=0.072, sum=0.191 (3)

min=0.917, mean=0.923, max=0.934, sum=2.768 (3)

min=0.922, mean=0.932, max=0.941, sum=2.797 (3)

min=0.682, mean=0.852, max=1.035, sum=2.555 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=4.915, mean=4.972, max=5, sum=14.915 (3)

min=853.851, mean=1281.577, max=1725.03, sum=3844.732 (3)

min=0.951, mean=0.956, max=0.962, sum=2.869 (3)

min=0.196, mean=0.213, max=0.234, sum=0.639 (3)

min=0.927, mean=0.932, max=0.936, sum=2.796 (3)

min=0.939, mean=0.946, max=0.951, sum=2.839 (3)

min=0.531, mean=0.637, max=0.757, sum=1.911 (3)

min=0.947, mean=0.953, max=0.957, sum=2.859 (3)

min=0.152, mean=0.158, max=0.166, sum=0.473 (3)

min=0.932, mean=0.941, max=0.948, sum=2.822 (3)

min=0.94, mean=0.946, max=0.95, sum=2.839 (3)

min=0.59, mean=0.732, max=0.881, sum=2.197 (3)

min=0.947, mean=0.957, max=0.964, sum=2.872 (3)

min=0.119, mean=0.136, max=0.165, sum=0.407 (3)

min=0.931, mean=0.947, max=0.955, sum=2.841 (3)

min=0.935, mean=0.95, max=0.959, sum=2.851 (3)

min=0.936, mean=0.938, max=0.943, sum=2.815 (3)

min=0.157, mean=0.182, max=0.199, sum=0.546 (3)

min=0.878, mean=0.896, max=0.916, sum=2.688 (3)

min=0.928, mean=0.933, max=0.937, sum=2.799 (3)

min=853.851, mean=1288.518, max=1745.851, sum=3865.553 (3)

min=0.926, mean=0.938, max=0.954, sum=2.814 (3)

min=0.088, mean=0.111, max=0.153, sum=0.333 (3)

min=0.915, mean=0.928, max=0.949, sum=2.784 (3)

min=0.92, mean=0.931, max=0.951, sum=2.792 (3)

min=0.952, mean=0.956, max=0.96, sum=2.869 (3)

min=0.153, mean=0.178, max=0.201, sum=0.534 (3)

min=0.935, mean=0.941, max=0.946, sum=2.822 (3)

min=0.941, mean=0.945, max=0.951, sum=2.835 (3)

min=0.931, mean=0.939, max=0.949, sum=2.818 (3)

min=0.187, mean=0.232, max=0.257, sum=0.695 (3)

min=0.864, mean=0.887, max=0.918, sum=2.662 (3)

min=0.902, mean=0.912, max=0.926, sum=2.737 (3)

min=2.908, mean=4.236, max=4.985, sum=12.708 (3)

min=1283.569, mean=1560.056, max=1777.712, sum=4680.167 (3)

min=0.944, mean=0.947, max=0.951, sum=2.842 (3)

min=0.177, mean=0.204, max=0.232, sum=0.612 (3)

min=0.913, mean=0.92, max=0.933, sum=2.76 (3)

min=0.93, mean=0.937, max=0.946, sum=2.811 (3)

min=0.957, mean=0.959, max=0.961, sum=2.878 (3)

min=0.137, mean=0.173, max=0.222, sum=0.519 (3)

min=0.931, mean=0.932, max=0.934, sum=2.797 (3)

min=0.948, mean=0.949, max=0.951, sum=2.848 (3)

min=0.924, mean=0.934, max=0.948, sum=2.802 (3)

min=0.01, mean=0.015, max=0.024, sum=0.045 (3)

min=0.921, mean=0.928, max=0.94, sum=2.783 (3)

min=0.918, mean=0.925, max=0.936, sum=2.775 (3)

min=0.714, mean=0.79, max=0.897, sum=2.37 (3)

min=1155.212, mean=1422.545, max=1836.212, sum=4267.636 (3)

min=1.002, mean=1.014, max=1.02, sum=3.042 (3)

min=0.936, mean=0.945, max=0.95, sum=2.836 (3)

min=0.305, mean=0.343, max=0.41, sum=1.029 (3)

min=0.907, mean=0.92, max=0.927, sum=2.761 (3)

min=0.927, mean=0.938, max=0.946, sum=2.814 (3)

min=3.425, mean=3.536, max=3.659, sum=10.608 (3)

min=4.876, mean=4.943, max=4.987, sum=14.83 (3)

min=1129.265, mean=1375.21, max=1727.698, sum=4125.631 (3)

T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.181, mean=0.207, max=0.26, sum=0.622 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.207, mean=0.291, max=0.36, sum=0.872 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.13, mean=0.17, max=0.227, sum=0.511 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.129, mean=0.168, max=0.22, sum=0.505 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.37, mean=0.393, max=0.436, sum=1.18 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=1000, mean=1000, max=1000, sum=3000 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=1.981, mean=2.44, max=3.074, sum=7.321 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.03, mean=0.03, max=0.03, sum=0.09 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=905.879, mean=910.174, max=913.752, sum=2730.521 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=5, mean=5, max=5, sum=15 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=3, mean=3, max=3, sum=9 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.941, mean=0.956, max=0.965, sum=2.867 (3)

min=0.05, mean=0.069, max=0.081, sum=0.206 (3)

min=0.907, mean=0.923, max=0.933, sum=2.768 (3)

min=0.93, mean=0.949, max=0.96, sum=2.846 (3)

min=0.709, mean=0.796, max=0.865, sum=2.389 (3)

min=2.903, mean=4.229, max=4.983, sum=12.688 (3)

min=1283.038, mean=1562.808, max=1784.2, sum=4688.425 (3)

min=0.929, mean=0.933, max=0.94, sum=2.8 (3)

min=0.098, mean=0.132, max=0.183, sum=0.396 (3)

min=0.895, mean=0.902, max=0.91, sum=2.706 (3)

min=0.912, mean=0.92, max=0.93, sum=2.759 (3)

min=0.479, mean=0.536, max=0.62, sum=1.607 (3)

min=4.846, mean=4.93, max=4.98, sum=14.79 (3)

min=1161.854, mean=1398.654, max=1747.025, sum=4195.961 (3)

min=0.917, mean=0.935, max=0.947, sum=2.805 (3)

min=0.335, mean=0.36, max=0.394, sum=1.08 (3)

min=0.878, mean=0.889, max=0.897, sum=2.666 (3)

min=0.896, mean=0.918, max=0.936, sum=2.753 (3)

min=0.404, mean=0.452, max=0.489, sum=1.355 (3)

min=1, mean=1.003, max=1.01, sum=3.01 (3)

min=0.53, mean=0.578, max=0.618, sum=1.735 (3)

min=0.085, mean=0.134, max=0.174, sum=0.401 (3)

min=0.447, mean=0.473, max=0.498, sum=1.418 (3)

min=0.49, mean=0.518, max=0.54, sum=1.554 (3)

min=0.414, mean=0.458, max=0.52, sum=1.373 (3)

min=0.941, mean=0.956, max=0.965, sum=2.868 (3)

min=0.05, mean=0.069, max=0.082, sum=0.207 (3)

min=0.907, mean=0.923, max=0.933, sum=2.769 (3)

min=0.931, mean=0.949, max=0.96, sum=2.847 (3)

min=0.917, mean=0.935, max=0.947, sum=2.804 (3)

min=0.335, mean=0.36, max=0.394, sum=1.079 (3)

min=0.878, mean=0.888, max=0.896, sum=2.665 (3)

min=0.896, mean=0.917, max=0.936, sum=2.752 (3)

min=0.959, mean=0.961, max=0.962, sum=2.882 (3)

min=0.011, mean=0.014, max=0.019, sum=0.043 (3)

min=0.917, mean=0.921, max=0.925, sum=2.762 (3)

min=0.946, mean=0.95, max=0.954, sum=2.851 (3)

min=2.89, mean=4.217, max=4.981, sum=12.652 (3)

min=1282.318, mean=1557.741, max=1776.111, sum=4673.222 (3)

min=0.955, mean=0.96, max=0.965, sum=2.881 (3)

min=0.011, mean=0.015, max=0.02, sum=0.045 (3)

min=0.929, mean=0.933, max=0.936, sum=2.799 (3)

min=0.951, mean=0.957, max=0.96, sum=2.871 (3)

min=0.932, mean=0.939, max=0.946, sum=2.816 (3)

min=0.285, mean=0.295, max=0.311, sum=0.884 (3)

min=0.895, mean=0.903, max=0.908, sum=2.709 (3)

min=0.92, mean=0.927, max=0.932, sum=2.782 (3)

min=0.63, mean=0.701, max=0.761, sum=2.104 (3)

min=4.846, mean=4.933, max=4.986, sum=14.798 (3)

min=1152.694, mean=1389.454, max=1744.631, sum=4168.363 (3)

min=0.946, mean=0.948, max=0.95, sum=2.844 (3)

min=0.189, mean=0.23, max=0.269, sum=0.69 (3)

min=0.906, mean=0.912, max=0.921, sum=2.736 (3)

min=0.925, mean=0.928, max=0.933, sum=2.785 (3)

min=0.748, mean=0.862, max=1.078, sum=2.586 (3)

min=4.842, mean=4.93, max=4.981, sum=14.789 (3)

min=1162.003, mean=1398.09, max=1750.717, sum=4194.271 (3)

min=0.248, mean=0.379, max=0.568, sum=1.137 (3)

min=0.108, mean=0.236, max=0.374, sum=0.707 (3)

min=0.17, mean=0.304, max=0.51, sum=0.911 (3)

min=0.162, mean=0.303, max=0.502, sum=0.91 (3)

min=0.276, mean=0.278, max=0.28, sum=0.834 (3)

min=0.33, mean=0.466, max=0.701, sum=1.397 (3)

min=0.172, mean=0.173, max=0.173, sum=0.518 (3)

min=391.442, mean=408.425, max=434.668, sum=1225.274 (3)

min=0.13, mean=0.337, max=0.556, sum=1.01 (3)

min=0.037, mean=0.225, max=0.41, sum=0.675 (3)

min=0.091, mean=0.276, max=0.485, sum=0.827 (3)

min=0.092, mean=0.271, max=0.484, sum=0.814 (3)

min=0.214, mean=0.215, max=0.217, sum=0.645 (3)

min=0.309, mean=0.449, max=0.689, sum=1.347 (3)

min=0.175, mean=0.176, max=0.176, sum=0.527 (3)

min=388.254, mean=407.098, max=435.686, sum=1221.293 (3)

min=0.932, mean=0.947, max=0.96, sum=2.842 (3)

min=0.164, mean=0.19, max=0.216, sum=0.569 (3)

min=0.904, mean=0.919, max=0.937, sum=2.756 (3)

min=0.929, mean=0.944, max=0.958, sum=2.831 (3)

min=1.488, mean=1.575, max=1.732, sum=4.724 (3)

min=0.906, mean=0.917, max=0.926, sum=2.752 (3)

min=0.289, mean=0.302, max=0.327, sum=0.905 (3)

min=0.872, mean=0.886, max=0.901, sum=2.659 (3)

min=0.898, mean=0.908, max=0.919, sum=2.725 (3)

min=0.515, mean=0.54, max=0.569, sum=1.62 (3)

min=0.939, mean=0.941, max=0.942, sum=2.822 (3)

min=0.065, mean=0.087, max=0.106, sum=0.262 (3)

min=0.92, mean=0.921, max=0.922, sum=2.763 (3)

min=0.933, mean=0.936, max=0.94, sum=2.807 (3)

min=4.845, mean=4.932, max=4.985, sum=14.796 (3)

min=1152.524, mean=1389.183, max=1743.988, sum=4167.55 (3)

min=0.923, mean=0.927, max=0.934, sum=2.782 (3)

min=0.093, mean=0.118, max=0.136, sum=0.355 (3)

min=0.883, mean=0.896, max=0.909, sum=2.687 (3)

min=0.904, mean=0.912, max=0.922, sum=2.737 (3)

min=0.925, mean=0.933, max=0.942, sum=2.8 (3)

min=0.104, mean=0.126, max=0.166, sum=0.378 (3)

min=0.855, mean=0.873, max=0.89, sum=2.62 (3)

min=0.917, mean=0.921, max=0.923, sum=2.762 (3)

min=0.218, mean=0.225, max=0.231, sum=0.676 (3)

min=2.916, mean=4.242, max=4.986, sum=12.726 (3)

min=1277.729, mean=1553.363, max=1768.607, sum=4660.089 (3)

min=0.831, mean=0.889, max=0.939, sum=2.668 (3)

min=0.231, mean=0.259, max=0.285, sum=0.776 (3)

min=0.716, mean=0.803, max=0.892, sum=2.41 (3)

min=0.792, mean=0.86, max=0.922, sum=2.581 (3)

min=0.105, mean=0.11, max=0.115, sum=0.331 (3)

min=0.5, mean=0.597, max=0.646, sum=1.792 (3)

min=0.161, mean=0.212, max=0.289, sum=0.637 (3)

min=0.476, mean=0.5, max=0.512, sum=1.5 (3)