By E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher RéSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2611-08417.git
Cite
Version link
https://openversions.com/index/arxiv-2611-08417/commit/f11887da1c22d14b33cb673e88d8d5c8e4c28cb3
Version identifier (commit hash)
f11887da1c22d14b33cb673e88d8d5c8e4c28cb3

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Applying a continuous-validation workflow to Terminal-Bench 2.0 surfaces defects in roughly 30% of the benchmark (28 tasks), and the resulting Terminal-Bench 2.1 repairs shift accuracy and close rankings while preserving broad rank order, whereas fixed resource budgets materially shape apparent agent performance and harness gaps.

Applying the continuous-validation audit workflow to Terminal-Bench 2.0 surfaced task-side defects in 28 tasks (roughly one third of the 89-task benchmark), comprising instruction-verification mismatches (57%), task drift (43%), and environment issues (29%).

Across 19 evaluated agents, Terminal-Bench 2.1 rankings remain highly correlated with Terminal-Bench 2.0 (Spearman ρ=0.958\rho = 0.958, 95% CI [0.881,0.979][0.881, 0.979]), while individual agent scores shift by up to 12 percentage points and leaderboard positions move by up to three ranks.

The largest accuracy gains in Terminal-Bench 2.1 are concentrated among tasks with instruction-verification mismatches, where five tasks increased from pass rates of at most 15% to over 50% and accounted for roughly 62.7% of the total absolute pass-rate change on the modified-task subset.

Fixed wall-clock and output-token budgets materially change apparent agent performance and harness gaps: extending the time limit up to 32×32\times closes a ∼20\sim 20 pp harness gap on GPT 5.4 and shrinks a ∼30\sim 30 pp gap on GPT 5.4 mini to ∼5\sim 5 pp.

Capable coding assistants (Claude Code Opus 4.6 and Codex GPT-5.4) agree on primary defect categories across all 89 Terminal-Bench 2.0 tasks at κ=0.80\kappa = 0.80, whereas lower-cost API auditors achieve κ=0.55–0.77\kappa = 0.55\text{--}0.77 on binary triage but low agreement on task drift (κ=0.21–0.41\kappa = 0.21\text{--}0.41).

The continuous-validation workflow transfers beyond Terminal-Bench, flagging 14 of 34 randomly selected tasks from an 80-task Harbor-Index v0.1 snapshot with 86% precision (12 confirmed defective by manual review).

Methods

A continuous-validation pipeline implemented with GitHub Actions consisting of an evidence collector agent, an auditor agent applying a four-category defect rubric (no defect, instruction-verification mismatch, task drift, environment issues), a verifier agent running deterministic /oracle checks, and propose-only diffs gated by human maintainer review.

Source excerpts

“For each task, the evidence agent produces a structured report containing information on the benchmark, the task specification, trace information, and community reports. ... An auditor applies the rubric to each task. ... A verifier reviews the evidence and the auditor’s label.” · Show in paper

“We implement this audit workflow with GitHub Actions that can be invoked manually or triggered automatically on pull requests for a given task. ... Each run is capped at five repair iterations, and the workflow grants pull-requests but prevents agents from directly changing repository state.” · Show in paper

Resource-issue repair convention setting binding resource axes (wall-clock time, memory, or CPU count) to the factor closest to the original proposal or at least 1.5x the observed requirement of the oracle solution and verifier.

Source excerpts

“For resource-related failures, we adjust the binding resource axes (wall-clock time, memory, or CPU count; sometimes more than one) to the factor closest to the original proposal or at least 1.5× the observed requirement of the oracle solution and verifier.” · Show in paper

Benchmark re-evaluation protocol comparing Terminal-Bench 2.0 and Terminal-Bench 2.1 across 19 agent-model pairs using the terminus-2 baseline harness and provider-preferred harnesses (claude-code, codex, gemini-cli), with Spearman rank correlation and task-bootstrap 95% confidence intervals.

Source excerpts

“We evaluate 19 agents from multiple providers and compare the accuracy in both TB 2.0 and TB 2.1, as illustrated in Fig. 4. The correlation is high (Spearman ρ = 0.958), with an OLS fit of TB2.1 = 2.1 + 1.04 · TB2.0.” · Show in paper

“Models were evaluated with a common baseline harness, terminus-2 [Merrill et al., 2026], and, where available, the model provider’s preferred agent harness. The latter includes Anthropic’s Claude Code CLI [Anthropic, 2025], OpenAI’s Codex CLI [OpenAI, 2025a, 2026d], and Google’s Gemini CLI [Google, 2025].” · Show in paper

Test-time scaling and resource-budget sweep increasing the wall-clock budget up to 32x the original task timeout across six agent-model pairs and within-vendor harness swaps (Codex vs. Terminus-2) to measure coverage and efficiency frontiers.

Source excerpts

“For each agent–model pair, we increase the wall-clock budget up to 32× the original task timeout and measure how many tasks are solved as the budget increases.” · Show in paper