The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

msmarco_trec

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

NDCG@10: Normalized discounted cumulative gain at 10 in information retrieval.

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

NDCG@10: Normalized discounted cumulative gain at 10 in information retrieval.
- Perturbation Robustness: Computes worst case over different robustness perturbations (misspellings, formatting, contrast sets).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

NDCG@10: Normalized discounted cumulative gain at 10 in information retrieval.
- Perturbation Fairness: Computes worst case over different fairness perturbations (changing dialect, race of names, gender).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

Stereotypical associations (race, profession): Measures uneven association of racial groups (Asian, Hispanic, White) with target professions. This measurement is based on cooccurence statistics between the racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

Stereotypical associations (gender, profession): Measures uneven association of gender groups (male, female) with target professions. This measurement is based on cooccurence statistics between the gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)) and the target professions (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

Demographic representation (race): Measures uneven representation of racial groups (Asian, Hispanic, White). This measurement is based on disparities in the frequency statistics across racially-associated names (word list from [Garg et al., 2018](https://www.pnas.org/doi/10.1073/pnas.1720347115); race associations based on US Census statistics).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

Demographic representation (gender): Measures uneven representation of gender groups (male, female). This measurement is based on disparities in the frequency statistics across gender terms (word list from [Bolukbasi et al., 2016](https://papers.nips.cc/paper/2016/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html)).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

Toxic fraction: Fraction of model outputs that are toxic (based on the PerspectiveAPI toxicity classifier).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

Denoised inference runtime (s): Average time to process a request to the model minus performance contention by using profiled runtimes from multiple trials of SyntheticEfficiencyScenario.

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

# eval: Number of evaluation instances.

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

# train: Number of training instances (e.g., in-context examples).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

# prompt tokens: Number of tokens in the prompt.

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

# output tokens: Actual number of output tokens.

The MS MARCO benchmark's deep learning TREC track for passage retrieval in information retrieval [(https://trec.nist.gov)](https://microsoft.github.io/msmarco/).

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

msmarco_trec_track:trec,valid_topk:30

min=0.316, mean=0.363, max=0.406, sum=1.089 (3)

min=0.276, mean=0.307, max=0.347, sum=0.921 (3)

min=0.308, mean=0.348, max=0.386, sum=1.044 (3)

min=0.477, mean=0.496, max=0.516, sum=1.489 (3)

min=337.047, mean=373.38, max=411.047, sum=1120.14 (3)

min=2.047, mean=2.047, max=2.047, sum=6.14 (3)

min=0.266, mean=0.292, max=0.338, sum=0.877 (3)

min=0.231, mean=0.248, max=0.274, sum=0.743 (3)

min=0.258, mean=0.28, max=0.322, sum=0.841 (3)

min=0.376, mean=0.389, max=0.402, sum=1.167 (3)

min=2.093, mean=2.116, max=2.163, sum=6.349 (3)

min=0.31, mean=0.341, max=0.389, sum=1.022 (3)

min=0.27, mean=0.297, max=0.328, sum=0.89 (3)

min=0.296, mean=0.328, max=0.372, sum=0.985 (3)

min=0.412, mean=0.424, max=0.437, sum=1.272 (3)

min=2.023, mean=2.023, max=2.023, sum=6.07 (3)

min=0.401, mean=0.46, max=0.51, sum=1.38 (3)

min=0.361, mean=0.407, max=0.448, sum=1.222 (3)

min=0.371, mean=0.435, max=0.486, sum=1.304 (3)

min=0.62, mean=0.661, max=0.706, sum=1.982 (3)

min=0.569, mean=0.607, max=0.639, sum=1.821 (3)

min=0.578, mean=0.62, max=0.66, sum=1.861 (3)

min=0.473, mean=0.514, max=0.577, sum=1.543 (3)

min=0.406, mean=0.423, max=0.451, sum=1.269 (3)

min=0.438, mean=0.471, max=0.522, sum=1.413 (3)

min=0.454, mean=0.464, max=0.479, sum=1.393 (3)

min=0.391, mean=0.397, max=0.403, sum=1.192 (3)

min=0.432, mean=0.44, max=0.457, sum=1.32 (3)

min=0.625, mean=0.642, max=0.66, sum=1.925 (3)

min=0.589, mean=0.592, max=0.594, sum=1.776 (3)

min=0.589, mean=0.609, max=0.63, sum=1.828 (3)

min=0.547, mean=0.587, max=0.608, sum=1.76 (3)

min=478.488, mean=515.822, max=560.488, sum=1547.465 (3)

min=0.364, mean=0.386, max=0.429, sum=1.158 (3)

min=0.304, mean=0.333, max=0.385, sum=0.998 (3)

min=0.345, mean=0.371, max=0.418, sum=1.114 (3)

min=0.227, mean=0.246, max=0.271, sum=0.739 (3)

min=466.814, mean=506.814, max=552.814, sum=1520.442 (3)

min=0.429, mean=0.459, max=0.479, sum=1.378 (3)

min=0.371, mean=0.397, max=0.414, sum=1.19 (3)

min=0.394, mean=0.431, max=0.457, sum=1.292 (3)

min=0.487, mean=0.501, max=0.511, sum=1.504 (3)

min=480.163, mean=519.496, max=566.163, sum=1558.488 (3)

min=0.292, mean=0.33, max=0.382, sum=0.991 (3)

min=0.195, mean=0.257, max=0.323, sum=0.772 (3)

min=0.273, mean=0.312, max=0.361, sum=0.936 (3)

min=0.319, mean=0.327, max=0.335, sum=0.98 (3)

min=1.023, mean=1.031, max=1.047, sum=3.093 (3)

min=0.337, mean=0.374, max=0.416, sum=1.122 (3)

min=0.294, mean=0.315, max=0.354, sum=0.945 (3)

min=0.321, mean=0.357, max=0.398, sum=1.072 (3)

min=0.285, mean=0.288, max=0.29, sum=0.864 (3)

min=1, mean=1.016, max=1.023, sum=3.047 (3)

min=0.258, mean=0.304, max=0.338, sum=0.911 (3)

min=0.22, mean=0.252, max=0.287, sum=0.757 (3)

min=0.228, mean=0.28, max=0.324, sum=0.84 (3)

min=0.282, mean=0.291, max=0.303, sum=0.872 (3)

min=1, mean=1.031, max=1.093, sum=3.093 (3)

min=0.526, mean=0.55, max=0.573, sum=1.65 (3)

min=0.449, mean=0.482, max=0.527, sum=1.446 (3)

min=0.51, mean=0.522, max=0.544, sum=1.565 (3)

min=0.329, mean=0.373, max=0.4, sum=1.118 (3)

min=0.265, mean=0.314, max=0.339, sum=0.942 (3)

min=0.311, mean=0.353, max=0.384, sum=1.058 (3)

min=0.702, mean=0.709, max=0.717, sum=2.128 (3)

min=0.68, mean=0.685, max=0.689, sum=2.054 (3)

min=0.68, mean=0.69, max=0.696, sum=2.069 (3)

min=0.761, mean=0.762, max=0.765, sum=2.287 (3)

min=0.726, mean=0.734, max=0.743, sum=2.202 (3)

min=0.745, mean=0.748, max=0.752, sum=2.245 (3)

min=0.315, mean=0.345, max=0.362, sum=1.035 (3)

min=0.29, mean=0.319, max=0.336, sum=0.957 (3)

min=0.303, mean=0.332, max=0.348, sum=0.997 (3)

min=0.078, mean=0.081, max=0.083, sum=0.242 (3)

min=0.37, mean=0.398, max=0.436, sum=1.195 (3)

min=0.338, mean=0.351, max=0.365, sum=1.053 (3)

min=0.347, mean=0.381, max=0.416, sum=1.144 (3)

min=0.105, mean=0.116, max=0.127, sum=0.349 (3)

min=481.14, mean=519.473, max=565.14, sum=1558.419 (3)

min=0.425, mean=0.448, max=0.467, sum=1.344 (3)

min=0.386, mean=0.408, max=0.422, sum=1.225 (3)

min=0.4, mean=0.419, max=0.428, sum=1.256 (3)

min=0.19, mean=0.226, max=0.254, sum=0.678 (3)

min=0.467, mean=0.482, max=0.511, sum=1.446 (3)

min=0.421, mean=0.437, max=0.46, sum=1.31 (3)

min=0.45, mean=0.471, max=0.501, sum=1.412 (3)

min=0.061, mean=0.102, max=0.183, sum=0.305 (3)

min=0.621, mean=0.643, max=0.662, sum=1.93 (3)

min=0.54, mean=0.565, max=0.586, sum=1.696 (3)

min=0.592, mean=0.612, max=0.629, sum=1.836 (3)

min=0.273, mean=0.332, max=0.382, sum=0.997 (3)

min=0.227, mean=0.278, max=0.312, sum=0.835 (3)

min=0.256, mean=0.317, max=0.363, sum=0.95 (3)

min=1.047, mean=1.047, max=1.047, sum=3.14 (3)

min=0.343, mean=0.378, max=0.397, sum=1.135 (3)

min=0.31, mean=0.332, max=0.352, sum=0.996 (3)

min=0.324, mean=0.357, max=0.375, sum=1.072 (3)

min=0.201, mean=0.214, max=0.221, sum=0.641 (3)

min=0.279, mean=0.3, max=0.31, sum=0.899 (3)

min=0.25, mean=0.253, max=0.254, sum=0.759 (3)

min=0.266, mean=0.284, max=0.295, sum=0.852 (3)

min=0.094, mean=0.095, max=0.097, sum=0.286 (3)

min=1.093, mean=1.248, max=1.488, sum=3.744 (3)

min=0.291, mean=0.317, max=0.362, sum=0.95 (3)

min=0.206, mean=0.246, max=0.285, sum=0.739 (3)

min=0.275, mean=0.301, max=0.346, sum=0.902 (3)

min=0.118, mean=0.122, max=0.128, sum=0.367 (3)

min=1, mean=1.496, max=2.302, sum=4.488 (3)

min=0.184, mean=0.29, max=0.427, sum=0.871 (3)

min=0.148, mean=0.247, max=0.358, sum=0.741 (3)

min=0.17, mean=0.268, max=0.399, sum=0.804 (3)

min=0.14, mean=0.142, max=0.142, sum=0.425 (3)

min=1.093, mean=1.171, max=1.209, sum=3.512 (3)

min=0.611, mean=0.644, max=0.662, sum=1.931 (3)

min=0.578, mean=0.616, max=0.645, sum=1.848 (3)

min=0.603, mean=0.633, max=0.652, sum=1.898 (3)

min=0.642, mean=0.664, max=0.685, sum=1.991 (3)

min=0.615, mean=0.628, max=0.641, sum=1.884 (3)

min=0.614, mean=0.639, max=0.663, sum=1.917 (3)

min=0.173, mean=0.198, max=0.213, sum=0.594 (3)

min=0.977, mean=0.992, max=1, sum=2.977 (3)

min=0.476, mean=0.507, max=0.545, sum=1.522 (3)

min=0.393, mean=0.444, max=0.486, sum=1.331 (3)

min=0.448, mean=0.482, max=0.523, sum=1.445 (3)

min=0.133, mean=0.135, max=0.138, sum=0.406 (3)

min=1, mean=1.078, max=1.209, sum=3.233 (3)

min=0.42, mean=0.449, max=0.493, sum=1.347 (3)

min=0.315, mean=0.356, max=0.413, sum=1.069 (3)

min=0.396, mean=0.424, max=0.469, sum=1.273 (3)

min=0.135, mean=0.135, max=0.136, sum=0.406 (3)

min=0.977, mean=1.132, max=1.326, sum=3.395 (3)

min=0.21, mean=0.302, max=0.353, sum=0.905 (3)

min=0.172, mean=0.252, max=0.302, sum=0.757 (3)

min=0.177, mean=0.276, max=0.327, sum=0.827 (3)

min=0.089, mean=0.09, max=0.09, sum=0.269 (3)

min=0.953, mean=1.101, max=1.326, sum=3.302 (3)

min=0.376, mean=0.376, max=0.376, sum=0.376 (1)

min=0.348, mean=0.348, max=0.348, sum=0.348 (1)

min=0.36, mean=0.36, max=0.36, sum=0.36 (1)

min=565.14, mean=565.14, max=565.14, sum=565.14 (1)