Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

copyright_code

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Longest common prefix length: Average length of longest common prefix between model generation and reference.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Edit distance (Levenshtein): Average Levenshtein edit distance between model generation and reference.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Edit similarity (Levenshtein): Average Levenshtein edit similarity (1 - distance normalized by length of longer sequence) between model generation and reference.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

# eval: Number of evaluation instances.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

# train: Number of training instances (e.g., in-context examples).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

# prompt tokens: Number of tokens in the prompt.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

# output tokens: Actual number of output tokens.

Scenario introduced in this work to measure copyright and memorization behavior for code, based off of [Carlini et al. (2021)](https://www.usenix.org/biblio-11958).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

min=0.055, mean=0.163, max=0.27, sum=0.325 (2)

min=6.389, mean=23.921, max=41.453, sum=47.842 (2)

min=0.003, mean=0.009, max=0.015, sum=0.018 (2)

min=0.641, mean=0.654, max=0.667, sum=1.308 (2)

min=25.204, mean=25.355, max=25.506, sum=50.71 (2)

min=1000, mean=1000, max=1000, sum=2000 (2)

min=23.255, mean=80.194, max=137.132, sum=160.387 (2)

min=1004.507, mean=1009.753, max=1014.999, sum=2019.506 (2)

min=0.048, mean=0.14, max=0.233, sum=0.281 (2)

min=6.304, mean=23.679, max=41.054, sum=47.359 (2)

min=0.003, mean=0.009, max=0.015, sum=0.017 (2)

min=0.333, mean=0.5, max=0.667, sum=1 (2)

min=0, mean=0.001, max=0.001, sum=0.001 (2)

min=8.153, mean=8.198, max=8.244, sum=16.397 (2)

min=1009.952, mean=1014.998, max=1020.044, sum=2029.996 (2)

Copyright (code)

copyright_code_datatag:prompt_num_line_1-min_lines_20.json

min=0.27, mean=0.27, max=0.27, sum=0.27 (1)

min=41.453, mean=41.453, max=41.453, sum=41.453 (1)

min=0.015, mean=0.015, max=0.015, sum=0.015 (1)

min=0.667, mean=0.667, max=0.667, sum=0.667 (1)

min=25.204, mean=25.204, max=25.204, sum=25.204 (1)

min=1000, mean=1000, max=1000, sum=1000 (1)

min=23.255, mean=23.255, max=23.255, sum=23.255 (1)

min=1004.507, mean=1004.507, max=1004.507, sum=1004.507 (1)

min=0.233, mean=0.233, max=0.233, sum=0.233 (1)

min=41.054, mean=41.054, max=41.054, sum=41.054 (1)

min=0.001, mean=0.001, max=0.001, sum=0.001 (1)

min=8.153, mean=8.153, max=8.153, sum=8.153 (1)

min=1009.952, mean=1009.952, max=1009.952, sum=1009.952 (1)

datatag: prompt_num_line_1-min_lines_20.json

copyright_code_datatag:prompt_num_line_10-min_lines_20.json

min=0.055, mean=0.055, max=0.055, sum=0.055 (1)

min=6.389, mean=6.389, max=6.389, sum=6.389 (1)

min=0.003, mean=0.003, max=0.003, sum=0.003 (1)

min=0.641, mean=0.641, max=0.641, sum=0.641 (1)

min=25.506, mean=25.506, max=25.506, sum=25.506 (1)

min=137.132, mean=137.132, max=137.132, sum=137.132 (1)

min=1014.999, mean=1014.999, max=1014.999, sum=1014.999 (1)

min=0.048, mean=0.048, max=0.048, sum=0.048 (1)

min=6.304, mean=6.304, max=6.304, sum=6.304 (1)

min=0.333, mean=0.333, max=0.333, sum=0.333 (1)

min=8.244, mean=8.244, max=8.244, sum=8.244 (1)

min=1020.044, mean=1020.044, max=1020.044, sum=1020.044 (1)