The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

summarization_xsum

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

ROUGE-2: Average ROUGE score [(Lin, 2004)](https://aclanthology.org/W04-1013/) based on 2-gram overlap.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

SummaC: Faithfulness scores based on the SummaC method of [Laban et al. (2022)](https://aclanthology.org/2022.tacl-1.10/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

QAFactEval: Faithfulness scores based on the SummaC method of [Laban et al. (2022)](https://aclanthology.org/2022.tacl-1.10/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

BERTScore (F1): Average BERTScore F1 [(Zhang et al., 2020)](https://openreview.net/pdf?id=SkeHuCVFDr) between model generation and reference summary.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Coverage: Extent to which the model-generated summaries are extractive fragments from the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Density: Extent to which the model-generated summaries are extractive summaries based on the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Compression: Extent to which the model-generated summaries are compressed relative to the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

HumanEval-faithfulness: Human evaluation score for faithfulness.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

HumanEval-relevance: Human evaluation score for relevance.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

HumanEval-coherence: Human evaluation score for coherence.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

# eval: Number of evaluation instances.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

# train: Number of training instances (e.g., in-context examples).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

# prompt tokens: Number of tokens in the prompt.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

# output tokens: Actual number of output tokens.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

summarization_xsum_dataset_name:xsum-sampled,sampling_min_length:50,sampling_max_length:150,doc_max_length:512

min=0.128, mean=0.129, max=0.131, sum=0.776 (6)

min=-0.294, mean=-0.287, max=-0.282, sum=-0.861 (3)

min=2.48, mean=3.182, max=3.598, sum=19.091 (6)

min=0.432, mean=0.435, max=0.438, sum=1.305 (3)

min=0.775, mean=0.784, max=0.792, sum=4.704 (6)

min=2.514, mean=2.63, max=2.802, sum=15.779 (6)

min=16.767, mean=16.862, max=16.987, sum=101.17 (6)

min=0.667, mean=0.667, max=0.667, sum=4.0 (6)

min=0.46, mean=0.472, max=0.483, sum=2.834 (6)

min=0.467, mean=0.48, max=0.505, sum=2.877 (6)

min=0.154, mean=0.186, max=0.216, sum=1.116 (6)

min=0, mean=0.002, max=0.004, sum=0.012 (6)

min=1.615, mean=1.629, max=1.648, sum=9.776 (6)

min=1099.388, mean=1133.388, max=1172.388, sum=6800.328 (6)

min=21.958, mean=22.013, max=22.106, sum=132.077 (6)

min=0.095, mean=0.102, max=0.107, sum=0.612 (6)

min=-0.26, mean=-0.239, max=-0.222, sum=-0.716 (3)

min=3.354, mean=3.675, max=4.009, sum=22.047 (6)

min=0.393, mean=0.4, max=0.405, sum=1.2 (3)

min=0.804, mean=0.808, max=0.813, sum=4.85 (6)

min=3.618, mean=3.757, max=3.939, sum=22.541 (6)

min=17.523, mean=18.133, max=18.761, sum=108.8 (6)

min=0.397, mean=0.424, max=0.451, sum=2.547 (6)

min=0.387, mean=0.426, max=0.467, sum=2.554 (6)

min=0.151, mean=0.172, max=0.189, sum=1.031 (6)

min=0.896, mean=0.903, max=0.91, sum=5.418 (6)

min=20.832, mean=21.299, max=21.809, sum=127.792 (6)

min=0.118, mean=0.122, max=0.127, sum=0.733 (6)

min=-0.282, mean=-0.272, max=-0.264, sum=-0.815 (3)

min=3.221, mean=3.447, max=3.575, sum=20.68 (6)

min=0.424, mean=0.429, max=0.434, sum=1.287 (3)

min=0.78, mean=0.783, max=0.785, sum=4.696 (6)

min=2.514, mean=2.64, max=2.767, sum=15.838 (6)

min=18.382, mean=19.012, max=19.445, sum=114.069 (6)

min=0.667, mean=0.667, max=0.667, sum=4 (6)

min=0.417, mean=0.442, max=0.485, sum=2.652 (6)

min=0.439, mean=0.557, max=0.667, sum=3.34 (6)

min=0.149, mean=0.171, max=0.211, sum=1.025 (6)

min=1.055, mean=1.07, max=1.082, sum=6.42 (6)

min=19.975, mean=20.468, max=21.141, sum=122.807 (6)

min=0.149, mean=0.152, max=0.157, sum=0.911 (6)

min=-0.298, mean=-0.282, max=-0.27, sum=-0.845 (3)

min=0.45, mean=0.454, max=0.458, sum=1.362 (3)

min=0.782, mean=0.786, max=0.79, sum=4.714 (6)

min=2.624, mean=2.816, max=3.113, sum=16.895 (6)

min=16.323, mean=16.857, max=17.149, sum=101.14 (6)

min=0.45, mean=0.465, max=0.474, sum=2.791 (6)

min=0.494, mean=0.522, max=0.536, sum=3.133 (6)

min=0.201, mean=0.214, max=0.224, sum=1.284 (6)

min=0, mean=0.001, max=0.002, sum=0.008 (6)

min=21.805, mean=22.092, max=22.577, sum=132.552 (6)

min=0.177, mean=0.182, max=0.186, sum=1.09 (6)

min=-0.325, mean=-0.32, max=-0.314, sum=-0.96 (3)

min=0.484, mean=0.489, max=0.493, sum=1.468 (3)

min=0.75, mean=0.755, max=0.761, sum=4.53 (6)

min=1.852, mean=2.145, max=2.331, sum=12.869 (6)

min=16.368, mean=16.588, max=16.809, sum=99.529 (6)

min=0.451, mean=0.466, max=0.478, sum=2.796 (6)

min=0.362, mean=0.399, max=0.429, sum=2.397 (6)

min=0.189, mean=0.205, max=0.224, sum=1.232 (6)

min=0.002, mean=0.003, max=0.004, sum=0.019 (6)

min=21.909, mean=22.142, max=22.392, sum=132.853 (6)

min=0.164, mean=0.167, max=0.173, sum=1.005 (6)

min=-0.31, mean=-0.289, max=-0.268, sum=-0.868 (3)

min=0.47, mean=0.475, max=0.48, sum=1.424 (3)

min=0.761, mean=0.766, max=0.77, sum=4.596 (6)

min=2.195, mean=2.359, max=2.464, sum=14.157 (6)

min=16.604, mean=17.044, max=17.299, sum=102.262 (6)

min=0.445, mean=0.456, max=0.463, sum=2.736 (6)

min=0.362, mean=0.466, max=0.532, sum=2.798 (6)

min=0.192, mean=0.207, max=0.233, sum=1.24 (6)

min=0, mean=0.001, max=0.002, sum=0.004 (6)

min=21.463, mean=21.75, max=22.241, sum=130.502 (6)

min=0.14, mean=0.142, max=0.145, sum=0.853 (6)

min=-0.306, mean=-0.278, max=-0.26, sum=-0.833 (3)

min=0.449, mean=0.45, max=0.451, sum=1.35 (3)

min=0.781, mean=0.782, max=0.783, sum=4.694 (6)

min=2.345, mean=2.659, max=2.826, sum=15.952 (6)

min=17.895, mean=18.03, max=18.262, sum=108.178 (6)

min=0.433, mean=0.464, max=0.492, sum=2.785 (6)

min=0.407, mean=0.58, max=0.667, sum=3.481 (6)

min=0.209, mean=0.22, max=0.234, sum=1.321 (6)

min=0.002, mean=0.002, max=0.002, sum=0.012 (6)

min=21.112, mean=21.228, max=21.315, sum=127.371 (6)

min=0.101, mean=0.105, max=0.107, sum=0.628 (6)

min=-0.217, mean=-0.213, max=-0.206, sum=-0.639 (3)

min=0.391, mean=0.394, max=0.396, sum=1.183 (3)

min=0.828, mean=0.834, max=0.838, sum=5.002 (6)

min=4.128, mean=4.393, max=4.529, sum=26.358 (6)

min=17.248, mean=17.535, max=17.956, sum=105.21 (6)

min=0.429, mean=0.442, max=0.453, sum=2.655 (6)

min=0.153, mean=0.165, max=0.183, sum=0.99 (6)

min=4.998, mean=4.999, max=5, sum=29.992 (6)

min=1472.903, mean=1532.912, max=1566.407, sum=9197.471 (6)

min=25.481, mean=26.021, max=26.315, sum=156.127 (6)

min=0.122, mean=0.124, max=0.126, sum=0.742 (6)

min=-0.233, mean=-0.225, max=-0.212, sum=-0.675 (3)

min=0.419, mean=0.423, max=0.427, sum=1.269 (3)

min=0.817, mean=0.818, max=0.819, sum=4.91 (6)

min=3.392, mean=3.507, max=3.668, sum=21.042 (6)

min=17.136, mean=17.376, max=17.524, sum=104.258 (6)

min=0.449, mean=0.45, max=0.451, sum=2.701 (6)

min=0.532, mean=0.547, max=0.565, sum=3.282 (6)

min=0.212, mean=0.214, max=0.217, sum=1.283 (6)

min=25.747, mean=25.987, max=26.212, sum=155.923 (6)

min=0.133, mean=0.136, max=0.14, sum=0.813 (6)

min=-0.251, mean=-0.241, max=-0.231, sum=-0.723 (3)

min=0.442, mean=0.444, max=0.446, sum=1.331 (3)

min=0.799, mean=0.807, max=0.816, sum=4.841 (6)

min=2.852, mean=3.08, max=3.225, sum=18.481 (6)

min=16.326, mean=16.97, max=17.573, sum=101.823 (6)

min=0.42, mean=0.439, max=0.456, sum=2.635 (6)

min=0.532, mean=0.544, max=0.556, sum=3.264 (6)

min=0.201, mean=0.206, max=0.21, sum=1.238 (6)

min=25.844, mean=26.423, max=26.988, sum=158.537 (6)

min=0.131, mean=0.134, max=0.137, sum=0.804 (6)

min=-0.278, mean=-0.271, max=-0.263, sum=-0.812 (3)

min=2.934, mean=3.066, max=3.179, sum=18.394 (6)

min=0.434, mean=0.437, max=0.441, sum=1.311 (3)

min=0.806, mean=0.808, max=0.811, sum=4.849 (6)

min=2.656, mean=2.691, max=2.726, sum=16.146 (6)

min=14.828, mean=15.182, max=15.567, sum=91.094 (6)

min=0.667, mean=0.778, max=0.889, sum=4.667 (6)

min=4.333, mean=4.398, max=4.444, sum=26.389 (6)

min=4.889, mean=4.898, max=4.917, sum=29.389 (6)

min=0.415, mean=0.439, max=0.454, sum=2.637 (6)

min=0.497, mean=0.541, max=0.59, sum=3.246 (6)

min=0.189, mean=0.207, max=0.22, sum=1.244 (6)

min=0.002, mean=0.004, max=0.006, sum=0.023 (6)

min=2.357, mean=2.408, max=2.45, sum=14.45 (6)

min=1456.402, mean=1510.735, max=1539.402, sum=9064.409 (6)

min=28.284, mean=28.94, max=29.546, sum=173.637 (6)

min=0.022, mean=0.03, max=0.038, sum=0.179 (6)

min=-0.365, mean=-0.35, max=-0.335, sum=-1.049 (3)

min=4.196, mean=4.778, max=5.107, sum=28.667 (6)

min=0.025, mean=0.059, max=0.095, sum=0.177 (3)

min=0.48, mean=0.515, max=0.553, sum=3.091 (6)

min=1.41, mean=1.764, max=2.014, sum=10.585 (6)

min=7.741, mean=8.934, max=10.222, sum=53.603 (6)

min=0.45, mean=0.467, max=0.5, sum=2.802 (6)

min=0.238, mean=0.309, max=0.356, sum=1.856 (6)

min=0.109, mean=0.172, max=0.212, sum=1.032 (6)

min=3.874, mean=3.9, max=3.923, sum=23.4 (6)

min=1456.338, mean=1501.338, max=1528.338, sum=9008.027 (6)

min=50.606, mean=54.066, max=57.05, sum=324.394 (6)

T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.07, mean=0.09, max=0.103, sum=0.539 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=-0.331, mean=-0.3, max=-0.268, sum=-0.901 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.083, mean=0.097, max=0.111, sum=0.292 (3)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.543, mean=0.579, max=0.605, sum=3.474 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=1.492, mean=1.684, max=1.861, sum=10.105 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=10.341, mean=11.178, max=11.672, sum=67.065 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.667, mean=0.667, max=0.667, sum=4 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.43, mean=0.444, max=0.463, sum=2.663 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.286, mean=0.457, max=0.617, sum=2.74 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.215, mean=0.27, max=0.328, sum=1.62 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0, mean=0.001, max=0.002, sum=0.004 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.523, mean=0.554, max=0.571, sum=3.326 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=518, mean=518, max=518, sum=3108 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=1.967, mean=2.068, max=2.214, sum=12.405 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.002, mean=0.01, max=0.019, sum=0.058 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=889.981, mean=907.769, max=929.006, sum=5446.614 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=64, mean=64, max=64, sum=384 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=3, mean=3, max=3, sum=18 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.125, mean=0.129, max=0.134, sum=0.775 (6)

min=-0.265, mean=-0.253, max=-0.236, sum=-0.758 (3)

min=2.761, mean=2.981, max=3.213, sum=17.888 (6)

min=0.431, mean=0.434, max=0.438, sum=1.301 (3)

min=0.794, mean=0.8, max=0.803, sum=4.797 (6)

min=2.71, mean=2.945, max=3.142, sum=17.67 (6)

min=18.323, mean=18.422, max=18.574, sum=110.533 (6)

min=0.638, mean=0.661, max=0.697, sum=3.968 (6)

min=4.212, mean=4.239, max=4.275, sum=25.431 (6)

min=4.773, mean=4.825, max=4.877, sum=28.952 (6)

min=0.456, mean=0.463, max=0.468, sum=2.78 (6)

min=0.532, mean=0.622, max=0.667, sum=3.73 (6)