Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

legal_support

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

Quasi-exact match: Fraction of instances that the predicted output matches a correct reference up to light processing.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

# eval: Number of evaluation instances.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

# train: Number of training instances (e.g., in-context examples).

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

# prompt tokens: Number of tokens in the prompt.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

# output tokens: Actual number of output tokens.

Scenario introduced in this work to measure fine-grained legal reasoning through reverse entailment.

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

legal_support_

min=0.483, mean=0.484, max=0.485, sum=1.452 (3)

min=0.481, mean=0.487, max=0.491, sum=1.46 (3)

min=428.902, mean=439.902, max=448.902, sum=1319.706 (3)

min=0.503, mean=0.514, max=0.532, sum=1.542 (3)

min=0.391, mean=0.395, max=0.398, sum=1.184 (3)

min=0.481, mean=0.504, max=0.544, sum=1.513 (3)

min=0.421, mean=0.425, max=0.428, sum=1.275 (3)

min=0.55, mean=0.562, max=0.573, sum=1.687 (3)

min=0.62, mean=0.639, max=0.656, sum=1.916 (3)

min=0.573, mean=0.575, max=0.577, sum=1.724 (3)

min=0.548, mean=0.558, max=0.569, sum=1.675 (3)

min=0.503, mean=0.513, max=0.519, sum=1.54 (3)

min=616.401, mean=629.734, max=640.401, sum=1889.202 (3)

min=0.495, mean=0.517, max=0.534, sum=1.55 (3)

min=0.517, mean=0.53, max=0.548, sum=1.589 (3)

min=0.622, mean=0.624, max=0.628, sum=1.873 (3)

min=0.552, mean=0.555, max=0.557, sum=1.665 (3)

min=590.753, mean=605.753, max=616.753, sum=1817.258 (3)

min=0.515, mean=0.543, max=0.558, sum=1.628 (3)

min=0.186, mean=0.193, max=0.204, sum=0.578 (3)

min=604.172, mean=620.172, max=634.172, sum=1860.515 (3)

min=0.597, mean=0.611, max=0.626, sum=1.834 (3)

min=0.145, mean=0.146, max=0.148, sum=0.439 (3)

min=669.509, mean=689.176, max=702.509, sum=2067.528 (3)

min=0.546, mean=0.558, max=0.573, sum=1.675 (3)

min=0.502, mean=0.507, max=0.51, sum=1.52 (3)

min=613.922, mean=629.922, max=641.922, sum=1889.767 (3)

min=0.479, mean=0.491, max=0.511, sum=1.474 (3)

min=0.346, mean=0.349, max=0.352, sum=1.048 (3)

min=0.501, mean=0.507, max=0.517, sum=1.521 (3)

min=0.298, mean=0.301, max=0.303, sum=0.903 (3)

min=0.513, mean=0.524, max=0.536, sum=1.571 (3)

min=0.31, mean=0.313, max=0.315, sum=0.938 (3)

min=0.493, mean=0.526, max=0.556, sum=1.577 (3)

min=0.476, mean=0.489, max=0.501, sum=1.468 (3)

min=0.56, mean=0.566, max=0.573, sum=1.699 (3)

min=0.599, mean=0.606, max=0.616, sum=1.818 (3)

min=0.47, mean=0.479, max=0.491, sum=1.436 (3)

min=0.066, mean=0.066, max=0.067, sum=0.199 (3)

min=0.503, mean=0.515, max=0.53, sum=1.544 (3)

min=0.245, mean=0.245, max=0.246, sum=0.736 (3)

min=597.329, mean=610.996, max=622.329, sum=1832.988 (3)

min=0.556, mean=0.558, max=0.56, sum=1.675 (3)

min=0.21, mean=0.211, max=0.212, sum=0.633 (3)

min=1.311, mean=1.323, max=1.348, sum=3.969 (3)

min=0.002, mean=0.002, max=0.002, sum=0.006 (3)

min=393.808, mean=395.955, max=400.249, sum=1187.865 (3)

min=0.503, mean=0.506, max=0.511, sum=1.517 (3)

min=0.18, mean=0.18, max=0.18, sum=0.54 (3)

min=1.274, mean=1.286, max=1.311, sum=3.859 (3)

min=389.967, mean=392.164, max=396.556, sum=1176.491 (3)

min=0.513, mean=0.532, max=0.546, sum=1.595 (3)

min=0.392, mean=0.425, max=0.443, sum=1.275 (3)

min=0.521, mean=0.527, max=0.538, sum=1.581 (3)

min=0.116, mean=0.118, max=0.12, sum=0.355 (3)

min=0.571, mean=0.58, max=0.595, sum=1.74 (3)

min=0.493, mean=0.504, max=0.513, sum=1.511 (3)

min=0.489, mean=0.496, max=0.505, sum=1.489 (3)

min=0.202, mean=0.203, max=0.203, sum=0.608 (3)

min=0.483, mean=0.49, max=0.501, sum=1.47 (3)

min=0.094, mean=0.094, max=0.095, sum=0.283 (3)

min=0.472, mean=0.492, max=0.517, sum=1.476 (3)

min=0.118, mean=0.118, max=0.118, sum=0.354 (3)

min=0.243, mean=0.372, max=0.444, sum=1.115 (3)

min=0.139, mean=0.14, max=0.14, sum=0.419 (3)

min=0.613, mean=0.622, max=0.626, sum=1.865 (3)

min=0.601, mean=0.615, max=0.63, sum=1.845 (3)

min=0.174, mean=0.174, max=0.174, sum=0.522 (3)

min=0.436, mean=0.442, max=0.452, sum=1.327 (3)

min=0.139, mean=0.139, max=0.139, sum=0.417 (3)

min=0.515, mean=0.517, max=0.521, sum=1.552 (3)

min=0.135, mean=0.135, max=0.135, sum=0.405 (3)

min=0.505, mean=0.515, max=0.532, sum=1.544 (3)

min=0.089, mean=0.089, max=0.089, sum=0.267 (3)

min=0.191, mean=0.193, max=0.195, sum=0.58 (3)

min=0.095, mean=0.095, max=0.095, sum=0.285 (3)

min=0.613, mean=0.624, max=0.63, sum=1.871 (3)

min=579.986, mean=597.652, max=607.986, sum=1792.957 (3)

min=1.014, mean=1.026, max=1.039, sum=3.078 (3)

min=0.513, mean=0.513, max=0.513, sum=0.513 (1)

min=597.329, mean=597.329, max=597.329, sum=597.329 (1)

min=0.448, mean=0.451, max=0.454, sum=1.352 (3)

min=0.228, mean=0.229, max=0.23, sum=0.687 (3)

min=586.591, mean=609.591, max=628.591, sum=1828.773 (3)

min=0.483, mean=0.492, max=0.505, sum=1.476 (3)

min=0.601, mean=0.623, max=0.636, sum=1.869 (3)

min=0.996, mean=0.996, max=0.996, sum=2.988 (3)

min=0.103, mean=0.105, max=0.106, sum=0.315 (3)

min=593.894, mean=604.56, max=616.894, sum=1813.681 (3)