Text classification is a general format that aims to classify text into a set of categories. This includes a wide range of classification tasks where the input is text.

miscellaneous_text_classification

How many models this model outperform on average (over columns).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

accuracy

min=0.225, mean=0.681, max=0.975, sum=22.475 (33)

min=0.15, mean=0.545, max=0.95, sum=18 (33)

min=0.2, mean=0.658, max=0.975, sum=21.7 (33)

min=0.225, mean=0.679, max=0.95, sum=22.4 (33)

min=0.225, mean=0.746, max=0.975, sum=24.625 (33)

min=0.225, mean=0.712, max=0.975, sum=23.5 (33)

min=0.25, mean=0.622, max=0.975, sum=20.525 (33)

min=0.025, mean=0.473, max=0.975, sum=15.625 (33)

min=0, mean=0.523, max=0.925, sum=17.25 (33)

min=0, mean=0.653, max=0.975, sum=21.55 (33)

min=0.225, mean=0.699, max=0.95, sum=23.075 (33)

min=0.25, mean=0.592, max=0.975, sum=19.525 (33)

min=0, mean=0.118, max=0.775, sum=3.9 (33)

min=0.1, mean=0.633, max=0.95, sum=20.875 (33)

min=0, mean=0.596, max=0.975, sum=19.675 (33)

min=0.125, mean=0.52, max=0.975, sum=17.15 (33)

min=0, mean=0.492, max=0.975, sum=16.225 (33)

min=0, mean=0.624, max=0.975, sum=20.6 (33)

min=0.1, mean=0.591, max=0.975, sum=19.5 (33)

min=0.05, mean=0.634, max=0.975, sum=20.925 (33)

min=0.025, mean=0.667, max=0.975, sum=22.0 (33)

min=0.275, mean=0.619, max=0.975, sum=20.425 (33)

min=0.025, mean=0.505, max=0.975, sum=16.65 (33)

min=0, mean=0.37, max=0.925, sum=12.2 (33)

min=0, mean=0.404, max=0.95, sum=13.325 (33)

min=0.075, mean=0.606, max=0.975, sum=20 (33)

min=0.175, mean=0.557, max=0.975, sum=18.375 (33)

min=0.025, mean=0.679, max=0.975, sum=22.4 (33)

min=0.025, mean=0.525, max=0.975, sum=17.325 (33)

min=0.1, mean=0.642, max=0.975, sum=21.2 (33)

min=0, mean=0.49, max=0.975, sum=16.175 (33)

min=0.025, mean=0.455, max=0.975, sum=15.025 (33)

min=0, mean=0.423, max=0.975, sum=13.975 (33)

min=0.075, mean=0.759, max=0.95, sum=25.05 (33)

min=0.15, mean=0.733, max=0.975, sum=24.175 (33)

min=0, mean=0.489, max=0.85, sum=16.15 (33)

min=0.125, mean=0.509, max=0.925, sum=16.8 (33)

min=0.05, mean=0.406, max=0.975, sum=13.4 (33)

min=0.3, mean=0.781, max=0.975, sum=25.775 (33)

min=0.225, mean=0.502, max=0.975, sum=5.525 (11)

min=0, mean=0.598, max=0.975, sum=19.725 (33)

min=0.275, mean=0.652, max=0.95, sum=21.5 (33)

min=0, mean=0.698, max=0.975, sum=23.05 (33)

min=0, mean=0.395, max=0.975, sum=13.05 (33)

Accuracy

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

10-bin expected calibration error: The average difference between the model's confidence and accuracy, averaged across 10 bins where each bin contains an equal number of points (only computed for classification tasks). Warning - not reliable for small datasets (e.g., with < 300 examples) because each bin will have very few examples.

calibration

min=0.103, mean=0.228, max=0.595, sum=7.528 (33)

min=0.134, mean=0.269, max=0.513, sum=8.875 (33)

min=0.113, mean=0.244, max=0.466, sum=8.048 (33)

min=0.095, mean=0.234, max=0.473, sum=7.733 (33)

min=0.126, mean=0.218, max=0.683, sum=7.184 (33)

min=0.119, mean=0.232, max=0.581, sum=7.664 (33)

min=0.128, mean=0.254, max=0.441, sum=8.368 (33)

min=0.115, mean=0.29, max=0.826, sum=9.575 (33)

min=0.143, mean=0.29, max=0.954, sum=9.577 (33)

min=0.072, mean=0.238, max=1, sum=7.863 (33)

min=0.071, mean=0.212, max=0.648, sum=7.002 (33)

min=0.246, mean=0.44, max=0.775, sum=14.508 (33)

min=0.0, mean=0.086, max=0.573, sum=2.84 (33)

min=0.093, mean=0.274, max=0.825, sum=9.044 (33)

min=0.115, mean=0.267, max=1, sum=8.804 (33)

min=0.151, mean=0.304, max=0.849, sum=10.027 (33)

min=0.084, mean=0.234, max=0.631, sum=7.714 (33)

min=0.078, mean=0.25, max=1, sum=8.255 (33)

min=0.11, mean=0.253, max=0.545, sum=8.337 (33)

min=0.05, mean=0.274, max=0.84, sum=9.055 (33)

min=0.041, mean=0.262, max=0.96, sum=8.637 (33)

min=0.116, mean=0.389, max=0.975, sum=12.832 (33)

min=0.067, mean=0.324, max=0.975, sum=10.705 (33)

min=0.0, mean=0.367, max=0.925, sum=12.1 (33)

min=0.0, mean=0.401, max=0.95, sum=13.228 (33)

min=0.1, mean=0.352, max=0.74, sum=11.606 (33)

min=0.126, mean=0.468, max=0.975, sum=15.455 (33)

min=0.089, mean=0.244, max=0.908, sum=8.049 (33)

min=0.103, mean=0.314, max=0.912, sum=10.346 (33)

min=0.073, mean=0.222, max=0.806, sum=7.328 (33)

min=0.054, mean=0.319, max=0.977, sum=10.54 (33)

min=0.1, mean=0.286, max=0.455, sum=9.428 (33)

min=0.066, mean=0.268, max=0.696, sum=8.86 (33)

min=0.037, mean=0.203, max=0.736, sum=6.696 (33)

min=0.043, mean=0.212, max=0.586, sum=6.999 (33)

min=0.079, mean=0.409, max=1, sum=13.49 (33)

min=0.102, mean=0.295, max=0.541, sum=9.737 (33)

min=0.018, mean=0.473, max=0.891, sum=15.613 (33)

min=0.025, mean=0.219, max=0.7, sum=7.225 (33)

min=0.225, mean=0.502, max=0.975, sum=5.524 (11)

min=0.045, mean=0.226, max=0.392, sum=7.451 (33)

min=0.05, mean=0.351, max=0.725, sum=11.575 (33)

min=0.025, mean=0.302, max=1, sum=9.95 (33)

min=0.011, mean=0.278, max=0.881, sum=9.176 (33)

Calibration

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

robustness

min=0.025, mean=0.555, max=0.875, sum=18.3 (33)

min=0.025, mean=0.443, max=0.95, sum=14.625 (33)

min=0.025, mean=0.513, max=0.775, sum=16.925 (33)

min=0.025, mean=0.555, max=0.925, sum=18.3 (33)

min=0.225, mean=0.69, max=0.95, sum=22.775 (33)

min=0.025, mean=0.618, max=0.875, sum=20.4 (33)

min=0, mean=0.498, max=0.975, sum=16.425 (33)

min=0, mean=0.402, max=0.975, sum=13.25 (33)

min=0, mean=0.436, max=0.825, sum=14.4 (33)

min=0, mean=0.564, max=0.975, sum=18.6 (33)

min=0.225, mean=0.6, max=0.95, sum=19.8 (33)

min=0.175, mean=0.527, max=0.95, sum=17.375 (33)

min=0, mean=0.085, max=0.775, sum=2.8 (33)

min=0, mean=0.563, max=0.925, sum=18.575 (33)

min=0, mean=0.385, max=0.975, sum=12.7 (33)

min=0, mean=0.403, max=0.975, sum=13.3 (33)

min=0, mean=0.489, max=0.925, sum=16.125 (33)

min=0.025, mean=0.502, max=0.975, sum=16.55 (33)

min=0, mean=0.552, max=0.975, sum=18.225 (33)

min=0, mean=0.599, max=0.975, sum=19.775 (33)

min=0.1, mean=0.53, max=0.975, sum=17.5 (33)

min=0, mean=0.399, max=0.975, sum=13.175 (33)

min=0, mean=0.331, max=0.875, sum=10.925 (33)

min=0, mean=0.349, max=0.95, sum=11.525 (33)

min=0, mean=0.48, max=0.975, sum=15.85 (33)

min=0.025, mean=0.405, max=0.85, sum=13.35 (33)

min=0, mean=0.545, max=0.85, sum=17.975 (33)

min=0, mean=0.445, max=0.95, sum=14.675 (33)

min=0, mean=0.413, max=0.975, sum=13.625 (33)

min=0, mean=0.409, max=0.975, sum=13.5 (33)

min=0, mean=0.345, max=0.975, sum=11.375 (33)

min=0.05, mean=0.714, max=0.95, sum=23.55 (33)

min=0, mean=0.666, max=0.975, sum=21.975 (33)

min=0, mean=0.399, max=0.775, sum=13.175 (33)

min=0.025, mean=0.383, max=0.925, sum=12.625 (33)

min=0, mean=0.335, max=0.925, sum=11.05 (33)

min=0.2, mean=0.708, max=0.975, sum=23.35 (33)

min=0, mean=0.427, max=0.975, sum=4.7 (11)

min=0, mean=0.577, max=0.975, sum=19.05 (33)

min=0.05, mean=0.518, max=0.95, sum=17.1 (33)

min=0, mean=0.669, max=0.975, sum=22.075 (33)

min=0, mean=0.211, max=0.65, sum=6.975 (33)

Robustness

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

fairness

min=0.125, mean=0.623, max=0.975, sum=20.55 (33)

min=0.15, mean=0.511, max=0.95, sum=16.85 (33)

min=0.175, mean=0.636, max=0.975, sum=21 (33)

min=0.2, mean=0.637, max=0.95, sum=21.025 (33)

min=0.225, mean=0.711, max=0.975, sum=23.45 (33)

min=0.2, mean=0.689, max=0.975, sum=22.725 (33)

min=0.2, mean=0.567, max=0.975, sum=18.725 (33)

min=0, mean=0.445, max=0.975, sum=14.7 (33)

min=0, mean=0.489, max=0.925, sum=16.15 (33)

min=0, mean=0.601, max=0.975, sum=19.825 (33)

min=0.225, mean=0.67, max=0.95, sum=22.1 (33)

min=0.2, mean=0.563, max=0.975, sum=18.575 (33)

min=0, mean=0.106, max=0.75, sum=3.5 (33)

min=0.05, mean=0.598, max=0.95, sum=19.75 (33)

min=0, mean=0.564, max=0.975, sum=18.625 (33)

min=0.125, mean=0.5, max=0.975, sum=16.5 (33)

min=0, mean=0.452, max=0.975, sum=14.9 (33)

min=0, mean=0.604, max=0.975, sum=19.925 (33)

min=0.1, mean=0.571, max=0.975, sum=18.85 (33)

min=0.05, mean=0.609, max=0.975, sum=20.1 (33)

min=0.025, mean=0.627, max=0.975, sum=20.7 (33)

min=0.275, mean=0.594, max=0.975, sum=19.6 (33)

min=0.025, mean=0.475, max=0.975, sum=15.675 (33)

min=0, mean=0.351, max=0.85, sum=11.575 (33)

min=0, mean=0.375, max=0.95, sum=12.375 (33)

min=0.075, mean=0.58, max=0.975, sum=19.125 (33)

min=0.175, mean=0.536, max=0.975, sum=17.7 (33)

min=0.025, mean=0.644, max=0.975, sum=21.25 (33)

min=0.05, mean=0.605, max=0.975, sum=19.95 (33)

min=0, mean=0.473, max=0.975, sum=15.625 (33)

min=0, mean=0.438, max=0.975, sum=14.45 (33)

min=0.05, mean=0.705, max=0.95, sum=23.275 (33)

min=0.125, mean=0.671, max=0.975, sum=22.15 (33)

min=0, mean=0.458, max=0.85, sum=15.125 (33)

min=0.125, mean=0.475, max=0.925, sum=15.675 (33)

min=0.05, mean=0.376, max=0.975, sum=12.4 (33)

min=0.025, mean=0.698, max=0.975, sum=23.05 (33)

min=0.175, mean=0.475, max=0.975, sum=5.225 (11)

min=0, mean=0.575, max=0.975, sum=18.975 (33)

min=0.25, mean=0.618, max=0.925, sum=20.4 (33)

min=0, mean=0.662, max=0.975, sum=21.85 (33)

min=0, mean=0.342, max=0.975, sum=11.3 (33)

Fairness

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

bias

min=0.125, mean=0.125, max=0.125, sum=0.375 (3)

min=0.079, mean=0.079, max=0.079, sum=0.237 (3)

Bias

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

toxicity

min=0, mean=0.001, max=0.025, sum=0.025 (33)

Toxicity

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks [(Alex et al., 2021)](https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/ca46c1b9512a7a8315fa3c5a946e8265-Abstract-round2.html).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.