Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

disinformation_reiteration

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Self-BLEU: Self-BLEU metric of [Zhu et al. (2018)](https://arxiv.org/pdf/1802.01886.pdf) as a measure of diversity in model generations.

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Entropy (Monte Carlo): Monte Carlo estimate of the entropy as a measure of diversity in model generations.

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

# eval: Number of evaluation instances.

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

# train: Number of training instances (e.g., in-context examples).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

# prompt tokens: Number of tokens in the prompt.

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

# output tokens: Actual number of output tokens.

Scenario from [Buchanan et al. (2021)](https://cset.georgetown.edu/publication/truth-lies-and-automation/) that tests the ability to reiterate disinformation content.

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=7.009, mean=8.027, max=10.263, sum=48.163 (6)

min=24.652, mean=26.658, max=29.865, sum=159.948 (6)

min=0.667, mean=0.667, max=0.667, sum=0.667 (1)

min=0.333, mean=0.556, max=0.667, sum=1.667 (3)

min=0.019, mean=0.225, max=0.5, sum=1.123 (5)

min=0, mean=0.006, max=0.019, sum=0.038 (6)

min=2.531, mean=2.786, max=2.94, sum=16.715 (6)

min=185.892, mean=197.462, max=206.032, sum=1184.772 (6)

min=53.161, mean=59.21, max=62.811, sum=355.259 (6)

min=6.484, mean=8.411, max=11.018, sum=50.465 (6)

min=25.45, mean=28.232, max=31.024, sum=169.389 (6)

min=0.333, mean=0.444, max=0.5, sum=1.333 (3)

min=0.079, mean=0.304, max=0.5, sum=1.521 (5)

min=0, mean=0.004, max=0.013, sum=0.025 (6)

min=1.099, mean=1.19, max=1.275, sum=7.139 (6)

min=51.919, mean=57.418, max=62.613, sum=344.511 (6)

min=6.636, mean=8.366, max=11.19, sum=50.197 (6)

min=25.132, mean=28.022, max=31.749, sum=168.13 (6)

min=0.667, mean=0.667, max=0.667, sum=1.333 (2)

min=0.433, mean=0.478, max=0.5, sum=1.433 (3)

min=0.137, mean=0.494, max=0.667, sum=2.471 (5)

min=0, mean=0.164, max=0.5, sum=0.987 (6)

min=0, mean=0.005, max=0.013, sum=0.03 (6)

min=1.556, mean=1.671, max=1.786, sum=10.026 (6)

min=54.243, mean=59.333, max=64.452, sum=356.001 (6)

min=7.18, mean=8.412, max=10.033, sum=50.473 (6)

min=23.502, mean=26.608, max=28.535, sum=159.646 (6)

min=0.667, mean=0.667, max=0.667, sum=2.667 (4)

min=0.333, mean=0.48, max=0.667, sum=2.4 (5)

min=0.029, mean=0.181, max=0.5, sum=1.087 (6)

min=0, mean=0.008, max=0.019, sum=0.047 (6)

min=55.216, mean=59.038, max=64.622, sum=354.226 (6)

min=6.469, mean=7.987, max=8.942, sum=47.921 (6)

min=23.113, mean=25.831, max=28.998, sum=154.985 (6)

min=0.417, mean=0.567, max=0.667, sum=2.833 (5)

min=0, mean=0.174, max=0.346, sum=0.871 (5)

min=54.838, mean=59.308, max=63.806, sum=355.846 (6)

min=7.121, mean=7.874, max=9.041, sum=47.243 (6)

min=24.581, mean=26.637, max=28.92, sum=159.82 (6)

min=0.438, mean=0.484, max=0.5, sum=1.938 (4)

min=0.333, mean=0.521, max=0.667, sum=2.607 (5)

min=0.025, mean=0.264, max=0.5, sum=1.581 (6)

min=0, mean=0.006, max=0.013, sum=0.037 (6)

min=53.351, mean=59.118, max=65.548, sum=354.709 (6)

min=6.569, mean=8.784, max=11.003, sum=52.705 (6)

min=25.743, mean=26.751, max=29.293, sum=160.503 (6)

min=0.389, mean=0.451, max=0.5, sum=1.806 (4)

min=0.333, mean=0.533, max=0.667, sum=2.133 (4)

min=0.093, mean=0.226, max=0.5, sum=1.129 (5)

min=0, mean=0.002, max=0.006, sum=0.013 (6)

min=53.387, mean=55.955, max=62.865, sum=335.731 (6)

min=96.379, mean=98.207, max=99.226, sum=589.243 (6)

min=13.533, mean=14.887, max=17.563, sum=89.321 (6)

min=0.431, mean=0.477, max=0.5, sum=1.431 (3)

min=0, mean=0.004, max=0.013, sum=0.026 (6)

min=239.27, mean=261.253, max=279.903, sum=1567.52 (6)

min=49.811, mean=60.969, max=90.645, sum=365.817 (6)

min=94.508, mean=95.872, max=97.898, sum=575.231 (6)

min=13.522, mean=14.935, max=16.425, sum=89.61 (6)

min=0.389, mean=0.434, max=0.5, sum=1.302 (3)

min=52.784, mean=60.563, max=66.806, sum=363.378 (6)

min=79.181, mean=84.677, max=88.36, sum=508.06 (6)

min=12.692, mean=14.364, max=17.705, sum=86.185 (6)

min=0.361, mean=0.454, max=0.5, sum=1.361 (3)

min=0, mean=0.006, max=0.013, sum=0.038 (6)

min=53.649, mean=59.193, max=74.161, sum=355.16 (6)

min=98.718, mean=99.348, max=100.0, sum=596.087 (6)

min=10.388, mean=11.393, max=12.503, sum=68.359 (6)

min=0, mean=0.001, max=0.006, sum=0.006 (6)

min=3.657, mean=3.761, max=3.977, sum=22.566 (6)

min=238.135, mean=266.025, max=293.581, sum=1596.147 (6)

min=55.351, mean=56.979, max=61.161, sum=341.874 (6)

min=8.217, mean=9.438, max=10.81, sum=56.626 (6)

min=37.149, mean=42.667, max=47.756, sum=256.002 (6)

min=0.389, mean=0.472, max=0.5, sum=1.889 (4)

min=0.333, mean=0.583, max=0.667, sum=2.333 (4)

min=0.077, mean=0.32, max=0.5, sum=1.921 (6)

min=3.243, mean=3.796, max=4.297, sum=22.777 (6)

min=232.774, mean=245.423, max=257.774, sum=1472.539 (6)

min=116.459, mean=131.675, max=150.419, sum=790.049 (6)

min=24.964, mean=47.722, max=60.262, sum=286.332 (6)

min=32.362, mean=54.935, max=84.355, sum=329.61 (6)

min=0.029, mean=0.331, max=0.5, sum=1.985 (6)

min=0, mean=0.003, max=0.006, sum=0.017 (6)

min=2.423, mean=2.432, max=2.434, sum=14.589 (6)

min=248.757, mean=284.389, max=312.355, sum=1706.335 (6)

min=99.542, mean=99.896, max=100.0, sum=599.379 (6)

min=12.311, mean=14.712, max=16.399, sum=88.275 (6)

min=0.667, mean=0.667, max=0.667, sum=2 (3)

min=0.167, mean=0.417, max=0.5, sum=1.667 (4)

min=2.454, mean=2.767, max=3.097, sum=16.605 (6)

min=240.162, mean=260.984, max=279.806, sum=1565.906 (6)

min=51.742, mean=59.392, max=67.27, sum=356.355 (6)

min=99.652, mean=99.911, max=100.0, sum=599.463 (6)

min=14.146, mean=15.515, max=17.63, sum=93.088 (6)

min=0.35, mean=0.463, max=0.5, sum=1.85 (4)

min=1.487, mean=1.663, max=1.856, sum=9.978 (6)

min=53.919, mean=61.691, max=70, sum=370.145 (6)

min=99.355, mean=99.892, max=100.0, sum=599.355 (6)

min=12.111, mean=14.174, max=15.625, sum=85.043 (6)

min=0.782, mean=0.897, max=1.036, sum=5.382 (6)

min=46.486, mean=55.744, max=66.935, sum=334.461 (6)

min=99.373, mean=99.896, max=100.0, sum=599.373 (6)

min=11.691, mean=14.033, max=21.203, sum=84.197 (6)

min=0.509, mean=0.567, max=0.697, sum=3.401 (6)

min=45.161, mean=54.24, max=75.194, sum=325.439 (6)

min=99.087, mean=99.695, max=100.0, sum=598.17 (6)

min=13.712, mean=14.425, max=14.778, sum=86.551 (6)

min=55.541, mean=60.803, max=65.351, sum=364.82 (6)

min=99.401, mean=99.9, max=100.0, sum=599.401 (6)

min=12.287, mean=14.01, max=17.298, sum=84.06 (6)

min=49.595, mean=56.649, max=64.968, sum=339.895 (6)

min=100.0, mean=100.0, max=100.0, sum=600.0 (6)

min=11.325, mean=13.254, max=14.858, sum=79.525 (6)

min=0.167, mean=0.347, max=0.5, sum=1.042 (3)

min=58.243, mean=64.893, max=77.581, sum=389.359 (6)

min=98.938, mean=99.542, max=100.0, sum=597.253 (6)

min=10.49, mean=12.264, max=13.618, sum=73.586 (6)

min=62.027, mean=66.057, max=70.676, sum=396.343 (6)

min=6.081, mean=8.264, max=9.972, sum=49.585 (6)

min=48.721, mean=55.554, max=67.67, sum=333.321 (6)

min=0.1, mean=0.261, max=0.5, sum=1.568 (6)

min=0, mean=0.002, max=0.006, sum=0.012 (6)

min=0.656, mean=0.8, max=1.02, sum=4.798 (6)

min=141.757, mean=173.718, max=222.419, sum=1042.31 (6)

min=7.569, mean=8.634, max=11.053, sum=51.805 (6)

min=47.897, mean=54.037, max=66.67, sum=324.224 (6)

min=0.25, mean=0.375, max=0.5, sum=1.5 (4)

min=0.1, mean=0.332, max=0.5, sum=1.658 (5)

min=0, mean=0.007, max=0.013, sum=0.042 (6)

min=0.528, mean=0.618, max=0.766, sum=3.71 (6)

min=241.216, mean=262.71, max=280.871, sum=1576.262 (6)

min=143.378, mean=169.728, max=215.968, sum=1018.37 (6)

min=48.594, mean=54.635, max=59.969, sum=327.81 (6)

min=91.22, mean=96.287, max=102.768, sum=577.725 (6)

min=0.464, mean=0.491, max=0.5, sum=1.964 (4)

min=0.667, mean=0.667, max=0.667, sum=3.333 (5)

min=0.192, mean=0.409, max=0.5, sum=2.452 (6)

min=0, mean=0.01, max=0.019, sum=0.057 (6)

min=5.224, mean=5.378, max=6.048, sum=32.268 (6)

min=36.215, mean=39.376, max=44.669, sum=236.255 (6)

min=114.58, mean=119.747, max=124.517, sum=718.482 (6)

min=0.283, mean=0.449, max=0.5, sum=2.246 (5)

min=0.333, mean=0.589, max=0.667, sum=3.532 (6)

min=0.065, mean=0.279, max=0.5, sum=1.674 (6)

min=0, mean=0.013, max=0.032, sum=0.081 (6)

min=3.397, mean=3.399, max=3.402, sum=20.394 (6)

min=252.757, mean=288.389, max=316.355, sum=1730.335 (6)

min=7.599, mean=8.861, max=10.295, sum=53.167 (6)

min=36.221, mean=40.761, max=46.187, sum=244.567 (6)

min=0.4, mean=0.475, max=0.5, sum=1.9 (4)

min=0.056, mean=0.329, max=0.5, sum=1.974 (6)

min=0.005, mean=0.012, max=0.032, sum=0.072 (6)

min=2.731, mean=3.4, max=4.062, sum=20.4 (6)

min=114.73, mean=140.629, max=160.484, sum=843.777 (6)

min=7.27, mean=10.93, max=13.997, sum=65.582 (6)

min=35.967, mean=41.689, max=49.139, sum=250.132 (6)

min=0.333, mean=0.417, max=0.667, sum=1.667 (4)

min=0, mean=0.258, max=0.5, sum=1.547 (6)

min=0, mean=0.004, max=0.011, sum=0.024 (6)

min=0.635, mean=0.805, max=1.007, sum=4.831 (6)

min=122.946, mean=150.91, max=193.774, sum=905.463 (6)

min=7.144, mean=9.855, max=11.997, sum=59.13 (6)

min=19.95, mean=23.04, max=25.495, sum=138.239 (6)

min=0.017, mean=0.299, max=0.5, sum=1.795 (6)

min=58.865, mean=69.271, max=76.742, sum=415.624 (6)

min=7.617, mean=9.453, max=11.818, sum=56.716 (6)

min=23.57, mean=26.463, max=28.404, sum=158.78 (6)

min=0.444, mean=0.593, max=0.667, sum=1.778 (3)

min=0.417, mean=0.472, max=0.5, sum=1.417 (3)

min=0.333, mean=0.467, max=0.667, sum=2.333 (5)

min=0, mean=0.277, max=0.5, sum=1.665 (6)

min=0, mean=0.002, max=0.013, sum=0.013 (6)

min=60.189, mean=70.856, max=80.387, sum=425.136 (6)

min=7.407, mean=9.38, max=12.913, sum=56.283 (6)

min=20.892, mean=25.178, max=27.843, sum=151.071 (6)

min=0.375, mean=0.475, max=0.5, sum=2.375 (5)

min=0.333, mean=0.5, max=0.667, sum=2 (4)

min=0.167, mean=0.351, max=0.5, sum=2.106 (6)

min=0, mean=0.008, max=0.013, sum=0.048 (6)

min=1.884, mean=2.279, max=2.68, sum=13.674 (6)

min=56.757, mean=70.004, max=83.387, sum=420.026 (6)

min=6.886, mean=8.37, max=10.414, sum=50.219 (6)

min=24.238, mean=28.157, max=32.645, sum=168.941 (6)

min=0.467, mean=0.567, max=0.667, sum=1.133 (2)