Targeted evaluation of training and inference efficiency.

efficiency

How many models this model outperform on average (over columns).

efficiency_detailed

Efficiency

Scenario introduced in this work to better understand inference runtime performance of various models.

# eval: Number of evaluation instances.

Scenario introduced in this work to better understand inference runtime performance of various models.

# train: Number of training instances (e.g., in-context examples).

Scenario introduced in this work to better understand inference runtime performance of various models.

truncated: Fraction of instances where the prompt itself was truncated (implies that there were no in-context examples).

Scenario introduced in this work to better understand inference runtime performance of various models.

# prompt tokens: Number of tokens in the prompt.

Scenario introduced in this work to better understand inference runtime performance of various models.

# output tokens: Actual number of output tokens.

Scenario introduced in this work to better understand inference runtime performance of various models.

# trials: Number of trials, where in each trial we choose an independent, random set of training instances.

general_information

min=1, mean=665.8, max=1536, sum=3329 (5)

min=13.6, mean=15.52, max=16, sum=77.6 (5)

min=21.7, mean=29.94, max=32, sum=149.7 (5)

min=37.7, mean=58.74, max=64, sum=293.7 (5)

min=1, mean=563.42, max=1024, sum=2817.1 (5)