The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

msmarco_regular

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

RR@10: Mean reciprocal rank at 10 in information retrieval.

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

RR@10: Mean reciprocal rank at 10 in information retrieval.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

RR@10: Mean reciprocal rank at 10 in information retrieval.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

# eval: Number of evaluation instances.

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

# train: Number of training instances (e.g., in-context examples).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

# prompt tokens: Number of tokens in the prompt.

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

# output tokens: Actual number of output tokens.

The MS MARCO benchmark's regular track for passage retrieval in information retrieval [(https://microsoft.github.io/msmarco/)](https://microsoft.github.io/msmarco/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

msmarco_regular_track:regular,valid_topk:30

min=0.173, mean=0.21, max=0.234, sum=0.63 (3)

min=0.131, mean=0.144, max=0.157, sum=0.433 (3)

min=0.151, mean=0.18, max=0.202, sum=0.54 (3)

min=0.482, mean=0.501, max=0.52, sum=1.502 (3)

min=1000, mean=1000, max=1000, sum=3000 (3)

min=349.303, mean=385.636, max=423.303, sum=1156.909 (3)

min=2, mean=2.001, max=2.004, sum=6.004 (3)

min=0.12, mean=0.147, max=0.178, sum=0.441 (3)

min=0.089, mean=0.105, max=0.128, sum=0.315 (3)

min=0.096, mean=0.117, max=0.143, sum=0.351 (3)

min=0.379, mean=0.393, max=0.406, sum=1.178 (3)

min=2.011, mean=2.072, max=2.163, sum=6.217 (3)

min=0.152, mean=0.161, max=0.179, sum=0.483 (3)

min=0.105, mean=0.121, max=0.133, sum=0.362 (3)

min=0.126, mean=0.138, max=0.155, sum=0.414 (3)

min=0.415, mean=0.428, max=0.44, sum=1.283 (3)

min=2.004, mean=2.011, max=2.023, sum=6.034 (3)

min=0.264, mean=0.285, max=0.316, sum=0.855 (3)

min=0.207, mean=0.222, max=0.244, sum=0.666 (3)

min=0.23, mean=0.253, max=0.284, sum=0.76 (3)

min=2.001, mean=2.009, max=2.02, sum=6.026 (3)

min=0.396, mean=0.398, max=0.401, sum=1.195 (3)

min=0.333, mean=0.337, max=0.343, sum=1.012 (3)

min=0.339, mean=0.342, max=0.346, sum=1.027 (3)

min=2, mean=2.001, max=2.003, sum=6.003 (3)

min=0.253, mean=0.293, max=0.322, sum=0.879 (3)

min=0.18, mean=0.227, max=0.253, sum=0.681 (3)

min=0.206, mean=0.243, max=0.271, sum=0.728 (3)

min=2.003, mean=2.006, max=2.008, sum=6.017 (3)

min=0.24, mean=0.247, max=0.257, sum=0.741 (3)

min=0.168, mean=0.177, max=0.186, sum=0.532 (3)

min=0.21, mean=0.215, max=0.221, sum=0.646 (3)

min=2.006, mean=2.012, max=2.022, sum=6.037 (3)

min=0.34, mean=0.382, max=0.407, sum=1.146 (3)

min=0.28, mean=0.308, max=0.326, sum=0.925 (3)

min=0.305, mean=0.345, max=0.369, sum=1.036 (3)

min=0.55, mean=0.578, max=0.599, sum=1.733 (3)

min=495.232, mean=532.565, max=577.232, sum=1597.696 (3)

min=1, mean=1.005, max=1.014, sum=3.014 (3)

min=0.212, mean=0.236, max=0.256, sum=0.709 (3)

min=0.158, mean=0.19, max=0.218, sum=0.57 (3)

min=0.189, mean=0.211, max=0.231, sum=0.633 (3)

min=0.246, mean=0.257, max=0.27, sum=0.77 (3)

min=484.472, mean=524.472, max=570.472, sum=1573.416 (3)

min=0.252, mean=0.273, max=0.294, sum=0.82 (3)

min=0.191, mean=0.207, max=0.223, sum=0.622 (3)

min=0.211, mean=0.233, max=0.251, sum=0.698 (3)

min=0.492, mean=0.499, max=0.504, sum=1.496 (3)

min=497.281, mean=536.614, max=583.281, sum=1609.843 (3)

min=0.164, mean=0.19, max=0.212, sum=0.571 (3)

min=0.109, mean=0.13, max=0.147, sum=0.39 (3)

min=0.136, mean=0.164, max=0.189, sum=0.493 (3)

min=0.322, mean=0.33, max=0.339, sum=0.989 (3)

min=1.008, mean=1.025, max=1.046, sum=3.074 (3)

min=0.143, mean=0.152, max=0.161, sum=0.456 (3)

min=0.101, mean=0.109, max=0.12, sum=0.326 (3)

min=0.126, mean=0.132, max=0.136, sum=0.396 (3)

min=0.286, mean=0.289, max=0.293, sum=0.867 (3)

min=1, mean=1.005, max=1.013, sum=3.014 (3)

min=0.291, mean=0.315, max=0.351, sum=0.945 (3)

min=0.201, mean=0.242, max=0.292, sum=0.725 (3)

min=0.239, mean=0.267, max=0.302, sum=0.802 (3)

min=1, mean=1.002, max=1.005, sum=3.005 (3)

min=0.147, mean=0.175, max=0.195, sum=0.525 (3)

min=0.11, mean=0.13, max=0.144, sum=0.389 (3)

min=0.123, mean=0.145, max=0.162, sum=0.436 (3)

min=1, mean=1.005, max=1.008, sum=3.015 (3)

min=0.423, mean=0.434, max=0.444, sum=1.301 (3)

min=0.372, mean=0.387, max=0.401, sum=1.161 (3)

min=0.402, mean=0.411, max=0.42, sum=1.232 (3)

min=0.467, mean=0.472, max=0.477, sum=1.416 (3)

min=0.429, mean=0.434, max=0.438, sum=1.303 (3)

min=0.444, mean=0.45, max=0.453, sum=1.35 (3)

min=0.127, mean=0.152, max=0.17, sum=0.457 (3)

min=0.094, mean=0.116, max=0.131, sum=0.349 (3)

min=0.106, mean=0.129, max=0.144, sum=0.387 (3)

min=0.081, mean=0.084, max=0.088, sum=0.252 (3)

min=0.171, mean=0.184, max=0.203, sum=0.551 (3)

min=0.082, mean=0.096, max=0.107, sum=0.288 (3)

min=0.137, mean=0.148, max=0.163, sum=0.445 (3)

min=0.111, mean=0.118, max=0.128, sum=0.355 (3)

min=499.575, mean=537.908, max=583.575, sum=1613.725 (3)

min=0.257, mean=0.288, max=0.316, sum=0.864 (3)

min=0.198, mean=0.235, max=0.263, sum=0.705 (3)

min=0.229, mean=0.26, max=0.288, sum=0.779 (3)

min=0.229, mean=0.241, max=0.262, sum=0.724 (3)

min=0.231, mean=0.237, max=0.243, sum=0.711 (3)

min=0.175, mean=0.179, max=0.187, sum=0.537 (3)

min=0.21, mean=0.214, max=0.221, sum=0.642 (3)

min=0.066, mean=0.076, max=0.089, sum=0.227 (3)

min=0.368, mean=0.377, max=0.394, sum=1.132 (3)

min=0.264, mean=0.287, max=0.315, sum=0.86 (3)

min=0.332, mean=0.341, max=0.354, sum=1.024 (3)

min=1.004, mean=1.011, max=1.02, sum=3.034 (3)

min=0.124, mean=0.158, max=0.185, sum=0.475 (3)

min=0.074, mean=0.105, max=0.125, sum=0.315 (3)

min=0.109, mean=0.14, max=0.166, sum=0.419 (3)

min=1.028, mean=1.067, max=1.136, sum=3.2 (3)

min=0.205, mean=0.211, max=0.221, sum=0.632 (3)

min=0.15, mean=0.154, max=0.157, sum=0.462 (3)

min=0.179, mean=0.185, max=0.192, sum=0.554 (3)

min=0.202, mean=0.211, max=0.218, sum=0.632 (3)

min=0.14, mean=0.162, max=0.189, sum=0.486 (3)

min=0.086, mean=0.11, max=0.14, sum=0.33 (3)

min=0.119, mean=0.14, max=0.167, sum=0.42 (3)

min=0.094, mean=0.094, max=0.095, sum=0.283 (3)

min=1.035, mean=1.112, max=1.183, sum=3.336 (3)

min=0.093, mean=0.122, max=0.145, sum=0.367 (3)

min=0.055, mean=0.073, max=0.086, sum=0.219 (3)

min=0.082, mean=0.105, max=0.123, sum=0.316 (3)

min=0.119, mean=0.122, max=0.126, sum=0.367 (3)

min=1.128, mean=1.537, max=2.075, sum=4.612 (3)

min=0.063, mean=0.102, max=0.156, sum=0.306 (3)

min=0.039, mean=0.072, max=0.111, sum=0.215 (3)

min=0.051, mean=0.086, max=0.134, sum=0.258 (3)

min=0.14, mean=0.142, max=0.143, sum=0.425 (3)

min=1.059, mean=1.219, max=1.379, sum=3.656 (3)

min=0.353, mean=0.368, max=0.389, sum=1.103 (3)

min=0.292, mean=0.304, max=0.319, sum=0.911 (3)

min=0.322, mean=0.335, max=0.353, sum=1.005 (3)

min=0.407, mean=0.421, max=0.437, sum=1.262 (3)

min=0.327, mean=0.344, max=0.366, sum=1.031 (3)

min=0.357, mean=0.373, max=0.39, sum=1.12 (3)

min=0.174, mean=0.192, max=0.207, sum=0.577 (3)

min=1.006, mean=1.014, max=1.024, sum=3.042 (3)

min=0.256, mean=0.271, max=0.287, sum=0.814 (3)

min=0.171, mean=0.198, max=0.222, sum=0.594 (3)

min=0.231, mean=0.244, max=0.26, sum=0.732 (3)

min=0.134, mean=0.136, max=0.138, sum=0.408 (3)

min=1.005, mean=1.031, max=1.08, sum=3.092 (3)

min=0.185, mean=0.208, max=0.25, sum=0.623 (3)

min=0.099, mean=0.122, max=0.16, sum=0.366 (3)

min=0.152, mean=0.174, max=0.213, sum=0.523 (3)

min=0.136, mean=0.136, max=0.136, sum=0.408 (3)

min=1.142, mean=1.212, max=1.282, sum=3.635 (3)

min=0.096, mean=0.134, max=0.158, sum=0.403 (3)

min=0.044, mean=0.069, max=0.091, sum=0.207 (3)

min=0.071, mean=0.107, max=0.133, sum=0.32 (3)

min=0.089, mean=0.09, max=0.091, sum=0.27 (3)

min=1.014, mean=1.123, max=1.303, sum=3.369 (3)

min=0.167, mean=0.167, max=0.167, sum=0.167 (1)

min=0.116, mean=0.116, max=0.116, sum=0.116 (1)

min=0.144, mean=0.144, max=0.144, sum=0.144 (1)

min=1000, mean=1000, max=1000, sum=1000 (1)

min=583.575, mean=583.575, max=583.575, sum=583.575 (1)