By Jessica McFadyen, Ole Jorgensen, Harry Coppock, Kevin Wei, Cozmin UdudecSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2606-17930.git
Cite
Version link
https://openversions.com/index/arxiv-2606-17930/commit/74dbb698e151d4d549a25f749bcf85e9261529b0
Version identifier (commit hash)
74dbb698e151d4d549a25f749bcf85e9261529b0

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2606.17930v3 OpenVersions: 4489c4d2-9d0c-4405-b69c-ea3c4a5b0a36

Posted

Claims

Findings

Headline finding

Benchmark scores for frontier LLMs are protocol-dependent, depending not only on the model being evaluated, but also on how much inference-time compute it is given and how that compute is allocated.

Inference scaling with larger token budgets is substantial but highly benchmark-dependent, with some benchmarks continuing to improve well beyond typical published budgets while others show weaker marginal gains under the tested protocol.

Typical published token budgets leave substantial additional performance unrealised on FrontierMath (+11.7 +/- 11.0 percentage points) and HLE (+11.9 +/- 10.4 percentage points), whereas SWE-Bench Pro (+0.3 +/- 0.3 pp), TerminalBench (+1.3 +/- 1.0 pp), and HealthBench (+0.3 +/- 0.4 pts) show little headroom under the tested serial scaling protocol.

Newer model generations usually achieve higher performance at large budgets, and these cross-generational gains are driven mainly by greater task reach and reliability rather than improved token efficiency.

Later model generations unlock a larger proportion of tasks on all six tested benchmarks, with reach gains concentrated on harder tasks on Cyber CTFs, FrontierMath, and TerminalBench.

Gains from inference scaling do not arise from one universal intervention; repeated submissions improve performance on all benchmarks, feedback matters most where it can guide continued search (HLE and SWE-Bench Pro), and parallel scaling is strongest on stateless benchmarks (HealthBench and HLE) and weakest on stateful ones.

Newer models generally benefit less from parallel sampling, with the gain from parallel sampling decreasing with model release date on all five benchmarks.

Methods

Fully crossed main experiment evaluating six frontier models across three generations (Opus 4/4.5/4.6 and GPT-5/5.2/5.4) on five benchmarks (TerminalBench 2.0, SWE-Bench Pro, FrontierMath, HealthBench Hard, and HLE exact-match) under two feedback conditions (no feedback and oracle score feedback), using a shared ReAct-style scaffold in Inspect AI with 5 independent trajectories per (benchmark, model, condition, task) cell, supplemented by two previously collected cyber benchmarks (Cyber CTFs and The Last Ones).

Source excerpts

“Our main experiment follows a fully crossed design with six frontier models (Table 1) evaluated on five non-cyber benchmarks under two feedback conditions, using one shared ReAct-style scaffold (Yao et al., 2023) implemented in Inspect AI (UK AI Security Institute, 2024a). For each task in each (benchmark, model, condition) cell, we run 5 independent trajectories.” · Show in paper

“We also include two previously collected cyber benchmarks (UK AI Security Institute, 2026a; Folkerts et al., 2026; UK AI Security Institute, 2026b) in the inference-scaling analyses only.” · Show in paper

Three uniform serial inference-scaling interventions: expanded per-trajectory total token budgets of 5M–30M tokens (with a separate 16,000 reasoning-token budget per generation call), summary-based context compaction triggered when running context exceeds 130k tokens, and iterative resubmission up to 999 submissions per trajectory with an LLM-judge repetition guard (GPT-4o-mini at temperature 0) that terminates trajectories after three or more semantically equivalent consecutive submissions.

Source excerpts

“We apply three deliberately simple inference-scaling techniques uniformly across every (benchmark, model, condition) cell: 1. Expanded total token budgets of 5M–30M tokens per trajectory (Table 2), sized to task complexity and one to three orders of magnitude above typical benchmark defaults. 2. Context compaction, which replaces earlier turns with a model-generated summary to enable serial scaling beyond the nominal context-window size. We adopt Inspect AI’s default summary-based strategy (UK AI Security Institute, 2024a), triggered when the running context exceeds 130k tokens (about 65% of the smallest 200k context window across models). 3. Iterative resubmission with a hard cap of 999 submissions per trajectory, allowing the model either to refine its previous answer or to try a substantially different approach between submissions. To reduce unproductive loops of near-identical submissions, we also employ a lightweight repetition guard that terminates a trajectory when a separate LLM judge finds three or more consecutive submissions semantically equivalent (Section A.1.3).” · Show in paper

Task-level decomposition regressions for reach, efficiency, and reliability: regressing the task unlock indicator (OLS with task-clustered standard errors), log10 median tokens-to-solve on unlocked cells (mixed-effects model with a task-level random intercept), and per-task solve rate on unlocked cells (OLS with task-clustered standard errors) on normalised model generation, leave-one-model-out task difficulty, and their interaction, plus a balanced-panel sensitivity check for efficiency.

Source excerpts

“For each benchmark, we fit a linear probability model regressing um,tu_{m,t} on model generation, leave-one-model-out task difficulty, and their interaction, by ordinary least squares (OLS) with cluster-robust standard errors at the task level.” · Show in paper

“For each benchmark, we regress log median tokens-to-solve on model generation, task difficulty, and their interaction, with a task-level random intercept.” · Show in paper

“For each benchmark, we regress per-task solve rate on model generation, leave-one-model-out task difficulty, and their interaction by OLS with cluster-robust standard errors at the task level.” · Show in paper

Parallel-versus-serial scaling evaluation using the unbiased pass@k estimator of Chen et al. (2021) at fixed total token budget B split across k in {1, 2, 5, 10} independent trajectories of B/k tokens each, alongside a judge reliability validation on a stratified subsample of HLE and HealthBench trajectories re-scored by GPT-5.4 and Claude Opus 4.6.

Source excerpts

“For total budget B, we estimate pass@k by splitting B across k independent trajectories of B/k tokens each and counting a task as solved if any trajectory’s first correct submission falls within budget. pass@k is computed task-wise using the unbiased estimator of Chen et al. (2021).” · Show in paper

“To bound this, we re-scored a stratified subsample of trajectories with two stronger judges, openai/gpt-5.4 and anthropic/claude-opus-4-6 , using the exact in-experiment grader template for each benchmark” · Show in paper