Headline finding
Benchmark scores for frontier LLMs are protocol-dependent, depending not only on the model being evaluated, but also on how much inference-time compute it is given and how that compute is allocated.
The claim is well-supported by the multi-benchmark experiments showing that token budgets, repeated submissions, feedback, and parallel vs. serial allocation all substantially affect measured performance.
“Overall, these findings suggest that frontier capability cannot be fully characterized by a single benchmark score measured under a single inference-time protocol. Observed performance depends not only on the model, but also on how much inference-time compute it is given and how that compute is allocated.” · Show in paper