Headline finding
Applying a continuous-validation workflow to Terminal-Bench 2.0 surfaces defects in roughly 30% of the benchmark (28 tasks), and the resulting Terminal-Bench 2.1 repairs shift accuracy and close rankings while preserving broad rank order, whereas fixed resource budgets materially shape apparent agent performance and harness gaps.
All components of the headline claim are directly supported by the reported audit counts (28/89 tasks), the 19-agent evaluation (Spearman rho = 0.958, rank shifts up to 3), and the 32x timeout sweep results.
“Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman ρ = 0.958). Beyond accuracy, we evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock and output-token budgets can materially change apparent agent performance” · Show in paper