In text summarization, given a piece of text (paragraph or document), the goal is to produce a much shorter summary.

summarization

How many models this model outperform on average (over columns).

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

ROUGE-2: Average ROUGE score [(Lin, 2004)](https://aclanthology.org/W04-1013/) based on 2-gram overlap.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

ROUGE-2: Average ROUGE score [(Lin, 2004)](https://aclanthology.org/W04-1013/) based on 2-gram overlap.

accuracy

min=0.137, mean=0.144, max=0.157, sum=0.861 (6)

min=0.128, mean=0.129, max=0.131, sum=0.776 (6)

min=0.123, mean=0.134, max=0.147, sum=0.802 (6)

min=0.095, mean=0.102, max=0.107, sum=0.612 (6)

min=0.127, mean=0.143, max=0.163, sum=0.859 (6)

min=0.118, mean=0.122, max=0.127, sum=0.733 (6)

min=0.14, mean=0.146, max=0.152, sum=0.875 (6)

min=0.149, mean=0.152, max=0.157, sum=0.911 (6)

min=0.142, mean=0.149, max=0.157, sum=0.892 (6)

min=0.177, mean=0.182, max=0.186, sum=1.09 (6)

min=0.131, mean=0.144, max=0.153, sum=0.865 (6)

min=0.164, mean=0.167, max=0.173, sum=1.005 (6)

min=0.122, mean=0.136, max=0.15, sum=0.813 (6)

min=0.14, mean=0.142, max=0.145, sum=0.853 (6)

min=0.048, mean=0.11, max=0.147, sum=0.661 (6)

min=0.101, mean=0.105, max=0.107, sum=0.628 (6)

min=0.117, mean=0.139, max=0.15, sum=0.834 (6)

min=0.122, mean=0.124, max=0.126, sum=0.742 (6)

min=0.133, mean=0.15, max=0.16, sum=0.899 (6)

min=0.133, mean=0.136, max=0.14, sum=0.813 (6)

min=0.142, mean=0.154, max=0.17, sum=0.927 (6)

min=0.131, mean=0.134, max=0.137, sum=0.804 (6)

min=0.052, mean=0.08, max=0.118, sum=0.478 (6)

min=0.022, mean=0.03, max=0.038, sum=0.179 (6)

min=0.121, mean=0.122, max=0.122, sum=0.73 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.07, mean=0.09, max=0.103, sum=0.539 (6)
☠ T0++ is explicitly trained on these datasets, i.e. data from the same distribution as the test set. See Table 5 on page 24 of https://arxiv.org/pdf/2110.08207.pdf.

min=0.14, mean=0.144, max=0.146, sum=0.861 (6)

min=0.125, mean=0.129, max=0.134, sum=0.775 (6)

min=0.115, mean=0.126, max=0.134, sum=0.758 (6)

min=0.106, mean=0.108, max=0.11, sum=0.649 (6)

min=0.03, mean=0.077, max=0.111, sum=0.459 (6)

min=0.086, mean=0.087, max=0.09, sum=0.524 (6)

min=0.031, mean=0.063, max=0.087, sum=0.377 (6)

min=0.031, mean=0.033, max=0.037, sum=0.199 (6)

min=0.153, mean=0.153, max=0.154, sum=0.92 (6)

min=0.148, mean=0.153, max=0.158, sum=0.919 (6)

min=0.116, mean=0.121, max=0.13, sum=0.728 (6)

min=0.095, mean=0.099, max=0.106, sum=0.596 (6)

min=0.15, mean=0.153, max=0.158, sum=0.919 (6)

min=0.122, mean=0.122, max=0.122, sum=0.73 (6)

min=0.156, mean=0.161, max=0.167, sum=0.966 (6)

min=0.147, mean=0.152, max=0.156, sum=0.913 (6)

min=0.127, mean=0.131, max=0.135, sum=0.787 (6)

min=0.093, mean=0.096, max=0.097, sum=0.573 (6)

min=0.108, mean=0.123, max=0.138, sum=0.738 (6)

min=0.098, mean=0.102, max=0.105, sum=0.61 (6)

min=0.043, mean=0.043, max=0.043, sum=0.257 (6)

min=0.008, mean=0.015, max=0.018, sum=0.087 (6)

min=0.03, mean=0.03, max=0.03, sum=0.182 (6)

min=0.049, mean=0.058, max=0.066, sum=0.345 (6)

min=0.132, mean=0.146, max=0.156, sum=0.875 (6)

min=0.153, mean=0.155, max=0.158, sum=0.929 (6)

min=0.119, mean=0.136, max=0.149, sum=0.816 (6)

min=0.123, mean=0.126, max=0.131, sum=0.757 (6)

min=0.151, mean=0.161, max=0.166, sum=0.966 (6)

min=0.162, mean=0.169, max=0.172, sum=1.013 (6)

min=0.139, mean=0.146, max=0.157, sum=0.877 (6)

min=0.107, mean=0.11, max=0.113, sum=0.661 (6)

min=0.087, mean=0.127, max=0.14, sum=0.889 (7)

min=0.045, mean=0.126, max=0.144, sum=0.884 (7)

min=0.038, mean=0.113, max=0.141, sum=0.789 (7)

min=0.035, mean=0.091, max=0.104, sum=0.636 (7)

min=0.016, mean=0.079, max=0.147, sum=0.472 (6)

min=0.041, mean=0.045, max=0.054, sum=0.273 (6)

min=0.046, mean=0.09, max=0.116, sum=0.541 (6)

min=0.012, mean=0.022, max=0.034, sum=0.134 (6)

min=0.151, mean=0.156, max=0.16, sum=0.935 (6)

min=0.122, mean=0.124, max=0.126, sum=0.744 (6)

min=0.148, mean=0.153, max=0.156, sum=1.074 (7)

min=0.087, mean=0.144, max=0.161, sum=1.006 (7)

min=0.144, mean=0.152, max=0.159, sum=1.061 (7)

min=0.056, mean=0.076, max=0.081, sum=0.533 (7)

min=0.147, mean=0.151, max=0.155, sum=0.907 (6)

min=0.044, mean=0.046, max=0.047, sum=0.275 (6)

min=0.134, mean=0.136, max=0.137, sum=0.813 (6)

min=0.034, mean=0.034, max=0.036, sum=0.206 (6)

min=0.169, mean=0.17, max=0.173, sum=1.022 (6)

min=0.142, mean=0.143, max=0.144, sum=0.858 (6)

min=0.114, mean=0.114, max=0.114, sum=0.227 (2)

min=0.079, mean=0.079, max=0.079, sum=0.158 (2)

min=0.144, mean=0.154, max=0.166, sum=0.926 (6)

min=0.131, mean=0.132, max=0.134, sum=0.794 (6)

min=0.142, mean=0.152, max=0.165, sum=0.91 (6)

min=0.1, mean=0.104, max=0.106, sum=0.624 (6)

min=0, mean=0.049, max=0.147, sum=0.294 (6)

min=0.144, mean=0.149, max=0.157, sum=0.895 (6)

min=0.009, mean=0.017, max=0.022, sum=0.103 (6)

min=0.019, mean=0.021, max=0.022, sum=0.124 (6)

Accuracy

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

SummaC: Faithfulness scores based on the SummaC method of [Laban et al. (2022)](https://aclanthology.org/2022.tacl-1.10/).

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

QAFactEval: Faithfulness scores based on the SummaC method of [Laban et al. (2022)](https://aclanthology.org/2022.tacl-1.10/).

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

BERTScore (F1): Average BERTScore F1 [(Zhang et al., 2020)](https://openreview.net/pdf?id=SkeHuCVFDr) between model generation and reference summary.

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

Coverage: Extent to which the model-generated summaries are extractive fragments from the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

Density: Extent to which the model-generated summaries are extractive summaries based on the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

Compression: Extent to which the model-generated summaries are compressed relative to the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

HumanEval-faithfulness: Human evaluation score for faithfulness.

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

HumanEval-relevance: Human evaluation score for relevance.

The CNN/DailyMail benchmark for text summarization ([Hermann et al., 2015](https://papers.nips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html); [Nallapati et al.,2016](https://aclanthology.org/K16-1028/)).

HumanEval-coherence: Human evaluation score for coherence.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

SummaC: Faithfulness scores based on the SummaC method of [Laban et al. (2022)](https://aclanthology.org/2022.tacl-1.10/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

QAFactEval: Faithfulness scores based on the SummaC method of [Laban et al. (2022)](https://aclanthology.org/2022.tacl-1.10/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

BERTScore (F1): Average BERTScore F1 [(Zhang et al., 2020)](https://openreview.net/pdf?id=SkeHuCVFDr) between model generation and reference summary.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Coverage: Extent to which the model-generated summaries are extractive fragments from the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Density: Extent to which the model-generated summaries are extractive summaries based on the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

Compression: Extent to which the model-generated summaries are compressed relative to the source document [(Grusky et al., 2018)](https://aclanthology.org/N18-1065/).

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

HumanEval-faithfulness: Human evaluation score for faithfulness.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

HumanEval-relevance: Human evaluation score for relevance.

The XSUM benchmark for text summarization of BBC news articles [(Narayan et al., 2018)](https://aclanthology.org/D18-1206/).

HumanEval-coherence: Human evaluation score for coherence.

summarization_metrics

min=0.491, mean=0.515, max=0.544, sum=1.545 (3)

min=4.661, mean=4.697, max=4.725, sum=28.182 (6)

min=0.264, mean=0.278, max=0.301, sum=0.834 (3)

min=0.965, mean=0.976, max=0.984, sum=5.856 (6)

min=40.605, mean=53.93, max=67.411, sum=323.578 (6)

min=8.981, mean=9.579, max=10.219, sum=57.476 (6)

min=-0.294, mean=-0.287, max=-0.282, sum=-0.861 (3)

min=2.48, mean=3.182, max=3.598, sum=19.091 (6)

min=0.432, mean=0.435, max=0.438, sum=1.305 (3)

min=0.775, mean=0.784, max=0.792, sum=4.704 (6)

min=2.514, mean=2.63, max=2.802, sum=15.779 (6)

min=16.767, mean=16.862, max=16.987, sum=101.17 (6)

min=0.488, mean=0.512, max=0.535, sum=1.537 (3)

min=4.664, mean=4.716, max=4.749, sum=28.295 (6)

min=0.229, mean=0.248, max=0.272, sum=0.745 (3)

min=0.971, mean=0.977, max=0.985, sum=5.861 (6)

min=55.528, mean=71.654, max=97.831, sum=429.924 (6)

min=5.872, mean=7.632, max=9.373, sum=45.79 (6)

min=-0.26, mean=-0.239, max=-0.222, sum=-0.716 (3)

min=3.354, mean=3.675, max=4.009, sum=22.047 (6)

min=0.393, mean=0.4, max=0.405, sum=1.2 (3)

min=0.804, mean=0.808, max=0.813, sum=4.85 (6)

min=3.618, mean=3.757, max=3.939, sum=22.541 (6)

min=17.523, mean=18.133, max=18.761, sum=108.8 (6)

min=0.514, mean=0.539, max=0.586, sum=1.617 (3)

min=4.706, mean=4.81, max=4.896, sum=28.859 (6)

min=0.247, mean=0.275, max=0.302, sum=0.824 (3)

min=0.966, mean=0.973, max=0.984, sum=5.84 (6)

min=31.118, mean=41.027, max=60.066, sum=246.163 (6)

min=8.092, mean=9.888, max=11.258, sum=59.326 (6)

min=-0.282, mean=-0.272, max=-0.264, sum=-0.815 (3)

min=3.221, mean=3.447, max=3.575, sum=20.68 (6)

min=0.424, mean=0.429, max=0.434, sum=1.287 (3)

min=0.78, mean=0.783, max=0.785, sum=4.696 (6)

min=2.514, mean=2.64, max=2.767, sum=15.838 (6)

min=18.382, mean=19.012, max=19.445, sum=114.069 (6)

min=0.533, mean=0.552, max=0.585, sum=1.655 (3)

min=0.273, mean=0.29, max=0.308, sum=0.871 (3)

min=0.965, mean=0.973, max=0.983, sum=5.838 (6)

min=18.643, mean=24.032, max=31.138, sum=144.19 (6)

min=10.389, mean=11.659, max=13.368, sum=69.956 (6)

min=-0.298, mean=-0.282, max=-0.27, sum=-0.845 (3)

min=0.45, mean=0.454, max=0.458, sum=1.362 (3)

min=0.782, mean=0.786, max=0.79, sum=4.714 (6)

min=2.624, mean=2.816, max=3.113, sum=16.895 (6)

min=16.323, mean=16.857, max=17.149, sum=101.14 (6)

min=0.442, mean=0.489, max=0.542, sum=1.466 (3)

min=0.299, mean=0.313, max=0.33, sum=0.94 (3)

min=0.952, mean=0.957, max=0.964, sum=5.745 (6)

min=12.535, mean=15.317, max=20.424, sum=91.904 (6)

min=11.809, mean=12.304, max=13.071, sum=73.825 (6)

min=-0.325, mean=-0.32, max=-0.314, sum=-0.96 (3)

min=0.484, mean=0.489, max=0.493, sum=1.468 (3)

min=0.75, mean=0.755, max=0.761, sum=4.53 (6)

min=1.852, mean=2.145, max=2.331, sum=12.869 (6)

min=16.368, mean=16.588, max=16.809, sum=99.529 (6)

min=0.469, mean=0.503, max=0.535, sum=1.508 (3)

min=0.281, mean=0.299, max=0.308, sum=0.896 (3)

min=0.953, mean=0.96, max=0.965, sum=5.76 (6)

min=14.681, mean=22.302, max=27.561, sum=133.811 (6)

min=10.404, mean=11.399, max=13.033, sum=68.391 (6)

min=-0.31, mean=-0.289, max=-0.268, sum=-0.868 (3)

min=0.47, mean=0.475, max=0.48, sum=1.424 (3)

min=0.761, mean=0.766, max=0.77, sum=4.596 (6)

min=2.195, mean=2.359, max=2.464, sum=14.157 (6)

min=16.604, mean=17.044, max=17.299, sum=102.262 (6)

min=0.466, mean=0.496, max=0.548, sum=1.488 (3)

min=0.242, mean=0.271, max=0.304, sum=0.812 (3)

min=0.952, mean=0.963, max=0.98, sum=5.779 (6)

min=15.279, mean=25.229, max=36.911, sum=151.374 (6)

min=9.923, mean=11.503, max=13.28, sum=69.017 (6)

min=-0.306, mean=-0.278, max=-0.26, sum=-0.833 (3)

min=0.449, mean=0.45, max=0.451, sum=1.35 (3)

min=0.781, mean=0.782, max=0.783, sum=4.694 (6)

min=2.345, mean=2.659, max=2.826, sum=15.952 (6)

min=17.895, mean=18.03, max=18.262, sum=108.178 (6)

min=-0.076, mean=0.32, max=0.527, sum=0.959 (3)

min=0.045, mean=0.188, max=0.278, sum=0.563 (3)

min=0.543, mean=0.834, max=0.982, sum=5.004 (6)

min=15.163, mean=35.663, max=51.192, sum=213.977 (6)

min=8.191, mean=9.346, max=11.345, sum=56.078 (6)

min=-0.217, mean=-0.213, max=-0.206, sum=-0.639 (3)

min=0.391, mean=0.394, max=0.396, sum=1.183 (3)

min=0.828, mean=0.834, max=0.838, sum=5.002 (6)

min=4.128, mean=4.393, max=4.529, sum=26.358 (6)

min=17.248, mean=17.535, max=17.956, sum=105.21 (6)

min=0.309, mean=0.481, max=0.569, sum=1.443 (3)