The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

lsat_qa

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

# eval: Number of evaluation instances.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

# train: Number of training instances (e.g., in-context examples).

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

# prompt tokens: Number of tokens in the prompt.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

# output tokens: Actual number of output tokens.

The LSAT benchmark for measuring analytical reasoning on the Law School Admission Test (LSAT; [Zhong et al., 2021](https://arxiv.org/pdf/2104.06598.pdf)).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

lsat_qa_task:all

min=0.217, mean=0.232, max=0.248, sum=0.696 (3)

min=0.639, mean=0.665, max=0.694, sum=1.996 (3)

min=845.539, mean=905.539, max=972.539, sum=2716.617 (3)

min=0.183, mean=0.196, max=0.204, sum=0.587 (3)

min=0.517, mean=0.536, max=0.556, sum=1.608 (3)

min=0.178, mean=0.188, max=0.2, sum=0.565 (3)

min=0.567, mean=0.587, max=0.61, sum=1.762 (3)

min=0.183, mean=0.191, max=0.204, sum=0.574 (3)

min=0.204, mean=0.219, max=0.239, sum=0.657 (3)

min=0.217, mean=0.22, max=0.226, sum=0.661 (3)

min=0.204, mean=0.217, max=0.235, sum=0.652 (3)

min=0.2, mean=0.235, max=0.261, sum=0.704 (3)

min=1135.557, mean=1180.557, max=1244.557, sum=3541.67 (3)

min=0.187, mean=0.188, max=0.191, sum=0.565 (3)

min=0.204, mean=0.212, max=0.222, sum=0.635 (3)

min=0.2, mean=0.213, max=0.23, sum=0.639 (3)

min=0.725, mean=0.736, max=0.752, sum=2.209 (3)

min=1144.796, mean=1190.129, max=1252.796, sum=3570.387 (3)

min=0.2, mean=0.209, max=0.217, sum=0.626 (3)

min=0.414, mean=0.418, max=0.424, sum=1.253 (3)

min=1120.117, mean=1156.451, max=1222.117, sum=3469.352 (3)

min=0.174, mean=0.186, max=0.204, sum=0.557 (3)

min=0.142, mean=0.142, max=0.142, sum=0.425 (3)

min=3.109, mean=3.609, max=3.891, sum=10.826 (3)

min=877.87, mean=926.957, max=952.543, sum=2780.87 (3)

min=0.191, mean=0.2, max=0.213, sum=0.6 (3)

min=0.669, mean=0.682, max=0.702, sum=2.047 (3)

min=1152.878, mean=1195.878, max=1258.878, sum=3587.635 (3)

min=0.183, mean=0.193, max=0.209, sum=0.578 (3)

min=0.477, mean=0.487, max=0.502, sum=1.462 (3)

min=0.204, mean=0.212, max=0.226, sum=0.635 (3)

min=0.382, mean=0.389, max=0.4, sum=1.168 (3)

min=0.183, mean=0.187, max=0.196, sum=0.561 (3)

min=0.416, mean=0.423, max=0.434, sum=1.27 (3)

min=0.191, mean=0.204, max=0.222, sum=0.613 (3)

min=0.2, mean=0.206, max=0.213, sum=0.617 (3)

min=0.174, mean=0.178, max=0.183, sum=0.535 (3)

min=0.226, mean=0.229, max=0.23, sum=0.687 (3)

min=0.143, mean=0.175, max=0.2, sum=0.526 (3)

min=0.134, mean=0.134, max=0.134, sum=0.402 (3)

min=0.152, mean=0.191, max=0.217, sum=0.574 (3)

min=0.252, mean=0.27, max=0.28, sum=0.81 (3)

min=1162.83, mean=1202.164, max=1266.83, sum=3606.491 (3)

min=0.148, mean=0.159, max=0.165, sum=0.478 (3)

min=0.221, mean=0.225, max=0.228, sum=0.675 (3)

min=0.891, mean=0.938, max=0.965, sum=2.813 (3)

min=0.039, mean=0.064, max=0.109, sum=0.191 (3)

min=406.374, mean=417.646, max=433.109, sum=1252.939 (3)

min=0.2, mean=0.207, max=0.213, sum=0.622 (3)

min=0.187, mean=0.19, max=0.195, sum=0.57 (3)

min=0.887, mean=0.93, max=0.957, sum=2.791 (3)

min=0.043, mean=0.07, max=0.113, sum=0.209 (3)

min=408.848, mean=420.317, max=436.187, sum=1260.952 (3)

min=0.191, mean=0.22, max=0.243, sum=0.661 (3)

min=0.334, mean=0.369, max=0.429, sum=1.106 (3)

min=0.165, mean=0.175, max=0.191, sum=0.526 (3)

min=0.093, mean=0.116, max=0.162, sum=0.348 (3)

min=0.222, mean=0.228, max=0.235, sum=0.683 (3)

min=0.161, mean=0.19, max=0.204, sum=0.57 (3)

min=0.183, mean=0.191, max=0.2, sum=0.574 (3)

min=0.214, mean=0.216, max=0.217, sum=0.647 (3)

min=0.178, mean=0.18, max=0.183, sum=0.539 (3)

min=0.102, mean=0.103, max=0.104, sum=0.309 (3)

min=0.252, mean=0.261, max=0.274, sum=0.783 (3)

min=0.124, mean=0.124, max=0.125, sum=0.373 (3)

min=0.161, mean=0.188, max=0.217, sum=0.565 (3)

min=0.14, mean=0.141, max=0.141, sum=0.422 (3)

min=0.226, mean=0.233, max=0.239, sum=0.7 (3)

min=0.213, mean=0.229, max=0.239, sum=0.687 (3)

min=0.211, mean=0.214, max=0.217, sum=0.641 (3)

min=0.17, mean=0.172, max=0.178, sum=0.517 (3)

min=0.14, mean=0.141, max=0.142, sum=0.422 (3)

min=0.183, mean=0.19, max=0.196, sum=0.57 (3)

min=0.148, mean=0.149, max=0.151, sum=0.447 (3)

min=0.209, mean=0.213, max=0.217, sum=0.639 (3)

min=0.101, mean=0.102, max=0.103, sum=0.305 (3)

min=0.2, mean=0.204, max=0.209, sum=0.612 (3)

min=0.101, mean=0.101, max=0.103, sum=0.304 (3)

min=0.243, mean=0.251, max=0.257, sum=0.752 (3)

min=1117.604, mean=1161.271, max=1231.604, sum=3483.813 (3)

min=1.004, mean=1.004, max=1.004, sum=3.013 (3)

min=0.213, mean=0.213, max=0.213, sum=0.213 (1)

min=1176.83, mean=1176.83, max=1176.83, sum=1176.83 (1)

min=0.174, mean=0.193, max=0.209, sum=0.578 (3)

min=0.564, mean=0.576, max=0.597, sum=1.728 (3)

min=1104.543, mean=1156.543, max=1215.543, sum=3469.63 (3)

min=0.178, mean=0.187, max=0.191, sum=0.561 (3)

min=0.222, mean=0.225, max=0.23, sum=0.674 (3)

min=0.974, mean=0.991, max=1, sum=2.974 (3)

min=0.23, mean=0.23, max=0.23, sum=0.691 (3)

min=0.247, mean=0.251, max=0.26, sum=0.754 (3)

min=1144.778, mean=1186.445, max=1242.778, sum=3559.335 (3)