Scenarios

The BoolQ benchmark for binary (yes/no) question answering (Clark et al., 2019).

The NarrativeQA benchmark for reading comprehension over narratives (Kočiský et al., 2017).

The NaturalQuestions (Kwiatkowski et al., 2019) benchmark for question answering based on naturally-occurring queries through Google Search. The input does not include the Wikipedia page with the answer.

The NaturalQuestions (Kwiatkowski et al., 2019) benchmark for question answering based on naturally-occurring queries through Google Search. The input includes the Wikipedia page with the answer.

The QuAC benchmark for question answering in the context of dialogues (Choi et al., 2018).

The HellaSwag benchmark for commonsense reasoning in question answering (Zellers et al., 2019).

The OpenbookQA benchmark for commonsense-intensive open book question answering (Mihaylov et al., 2018).

The TruthfulQA benchmarking for measuring model truthfulness and commonsense knowledge in question answering (Lin et al., 2022).

The Massive Multitask Language Understanding (MMLU) benchmark for knowledge-intensive question answering across 57 domains (Hendrycks et al., 2021).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval (https://microsoft.github.io/msmarco/).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval (https://trec.nist.gov).

The CNN/DailyMail benchmark for text summarization (Hermann et al., 2015; Nallapati et al.,2016).

The XSUM benchmark for text summarization of BBC news articles (Narayan et al., 2018).

The IMDB benchmark for sentiment analysis in movie review (Maas et al., 2011).

The Real-world annotated few-shot (RAFT) meta-benchmark of 11 real-world text classification tasks (Alex et al., 2021).

The CivilComments benchmark for toxicity detection (Borkan et al., 2019).

The International Corpus of English (ICE) drawn from English speakers from various places in the world, initiated by Greenbaum (1991).

The Pile corpus for measuring lanugage model performance across various domains (Gao et al., 2020).

The TwitterAAE corpus of Blodgett et al. (2016) for measuring language model performance in tweets as a function of speaker dialect.

The TwitterAAE corpus of Blodgett et al. (2016) for measuring language model performance on African-American-aligned Tweets.

The TwitterAAE corpus of Blodgett et al. (2016) for measuring language model performance on White-aligned Tweets.

The Benchmark of Linguistic Minimal Pairs for English (BLiMP) for measuring performance on linguistic phenomena using minimal pair design (Warstadt et al., 2020).

Scenario introduced in this work, inspired by Petroni et al. (2019), to more extensively test factual knowledge.

The bAbI benchmark for measuring understanding and reasoning (Weston et al., 2015).

Scenario testing hierarchical reasoning through the Dyck formal languages (Suzgun et al., 2019).

Scenario introduced in this work to test numerical reasoning via symbolic regression.

Synthetic reasoning tasks defined using abstract symbols based on LIME (Wu et al., 2021).

Synthetic reasoning tasks defined using simple natural language based on LIME (Wu et al., 2021).

The grade school math word problems dataset (GSM8K) for testing mathematical reasoning on grade-school math problems (Cobbe et al., 2021).

The MATH benchmark for measuring mathematical problem solving on competition math problems (Hendrycks et al., 2021).

The MATH benchmark for measuring mathematical problem solving on competition math problems with chain-of-thoughts style reasoning (Hendrycks et al., 2021).

The APPS benchmark for measuring competence on code challenges (Hendrycks et al., 2021).

The HumanEval benchmark for measuring functional correctness for synthesizing programs from docstrings (Chen et al., 2021).

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; Zhong et al., 2021).

The BillSum benchmark for legal text summarization (Kornilova & Eidelmann, 2020).

The MultiLexSum benchmark for legal text summarization (Shen et al., 2022).

The EurLexSum benchmark for legal text summarization (Aumiller et al., 2022).

Scenario from Mei et al. (2021) that tests the ability to impute missing entities in a data table.

Scenario from Magellan (Konda et al., 2016) that tests the ability to determine if two entities match.

Scenario introduced in this work to measure copyright and memorization behavior for books, based off of Carlini et al. (2021).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of Carlini et al. (2021).

Scenario from Buchanan et al. (2021) that tests the ability to reiterate disinformation content.

Scenario from Buchanan et al. (2021) that tests the ability to generate divisive and wedging content.

The Bias Benchmark for Question Answering (BBQ) for measuring social bias in question answering in ambiguous and unambigous context (Parrish et al., 2022).

The Bias in Open-Ended Language Generation Dataset (BOLD) for measuring biases and toxicity in open-ended language generation (Dhamala et al., 2021).

The RealToxicityPrompts dataset for measuring toxicity in prompted model generations (Gehman et al., 2020).

Scenario introduced in this work to better understand inference runtime performance of various models.