By Soham Dan, Jeff Da, Sanyam Bhutani, Miguel Romero Calvo, Ivan Guevara, Fernando Carabedo, Jose Bernal, Julio Reyna, Luis Leandro Monsalve Mendez, Ying Liu, Daniel Yue ZhangSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2610-24206.git
Cite
Version link
https://openversions.com/index/arxiv-2610-24206/commit/5cb9e448eea53053ded19e91b4b2654b21b0dc58
Version identifier (commit hash)
5cb9e448eea53053ded19e91b4b2654b21b0dc58

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2610.24206v2 OpenVersions: 69c96b9d-fa8a-4de2-969c-8f960d00759a

Posted

Claims

Findings

Headline finding

SWE-Bench Pro V2 repairs or retires 480 of the 731 original SWE-Bench Pro tasks via agentic audits, expert adjudication, blind re-implementation, and verifier hardening, resulting in a 642-task public split and a 51-task Hard split evaluated under a hardened protocol.

On the 642-task SWE-Bench Pro V2 split, frontier agents resolve up to 99.4% of tasks while other systems stay clearly separated (87.2%–99.4% overall), and the 51-task Hard split spreads systems from 52.9% to 98.0%, though it does not separate the top four systems.

Every task in the assembled 642-task SWE-Bench Pro V2 set is solvable: the reference patch passes every task, the empty patch fails every task, and each task is solved by at least one frontier model.

The locked evaluation protocol and clean-container re-grading prevent network-based solution retrieval and out-of-patch dependency forgeries that in-sandbox grading admits.

For open-weight models GLM-5.3 and Kimi K3, the agent scaffold moves token cost more than accuracy, whereas Inkling's accuracy varies by 5.8–7.5 percentage points across scaffolds.

Across all 642 tasks attempted by the three open-weight models in three scaffolds, task identity accounts for 31.7% of the variance in solved-or-failed, the model accounts for 2.65%, and the scaffold accounts for 0.29%.

Testing after the last edit roughly doubles the odds of solving for GLM-5.3, Kimi K3, and Inkling when scaffold, language, and run length are held constant, and the median failed run writes 1.6–2.0 times the completion tokens of the median solved run in the same model–scaffold pair.

Methods

Three-round task repair pipeline combining container image fixes, automated audits, two agentic audits (artifact audit and failure audit), 23-engineer adjudication over 404 hours, blind re-implementation validation, and a final cross-check review.

Source excerpts

“Figure 2 traces the tasks through three repair rounds, an engineer review and a final review. Round 1 repaired 40 container images whose reference patches carried test and fixture files or whose instructions lacked the standard template. Round 2 ran an automated audit of difficulty calibration, instruction–test alignment and reward hacking; it flagged 387 tasks (53%), found 326 clean and errored on 18.” · Show in paper

“Twenty-three contracted engineers adjudicated every flag in 404 hours.” · Show in paper

“Repairs were validated blind: a second engineer who had seen neither the original task nor the reference patch implemented the edited instruction, and the edit shipped only if that implementation passed the hidden tests.” · Show in paper

Verifier hardening across all tasks, including scoping Go tests to graded packages, capping pytest-xdist workers at four, fixing jest parser test-title truncation and file assignment, and resetting test files, fixtures, and Python bytecode caches before running hidden tests.

Source excerpts

“Scoping it to the graded packages on 253 tasks removed out-of-memory failures and timeouts. Run scripts on 231 tasks that executed the full suite were restricted to the graded tests. With automatic worker counts, pytest-xdist spawned up to 128 workers on small test files and crashed with “no tests ran”: we capped it at four workers on 239 tasks and inside ansible-test on 83.” · Show in paper

“Before the hidden tests run, the verifier (i) restores every path the test patch touches and deletes only paths it creates; (ii) restores modified tracked test data (test data, fixtures, mocks, snapshots), with allowlists for the six reference patches that change fixtures; (iii) removes agent-written test files that collide with hidden tests; (iv) clears stale git lock files left by killed agent commands; and (v) deletes __pycache__ and .pyc files.” · Show in paper

Locked evaluation protocol that restricts network access during the agent phase to the model endpoint alone, disables web tools, scrubs git history, and re-grades the staged git diff in a fresh container (with a majority-of-three tie-breaker if sandbox and clean grades disagree) alongside trajectory and patch audits.

Source excerpts

“Under the lock, the sandbox can reach only the model endpoint during the agent phase (the allowlist names that hostname alone), web tools that scaffolds execute outside the sandbox are disabled, and the images carry no fixing commit in their git history.” · Show in paper

“We now save the agent’s changes as a patch, apply it to a fresh copy of the task image, run the unchanged verifier there and report that grade, so that binaries or module caches do not carry over.” · Show in paper

“When the sandbox grade and the clean grade disagree, two more clean grades are taken and the majority of the three counts.” · Show in paper

Hard split selection by filtering for tasks where at least two of five model families fail across runs, followed by expert review, yielding 51 tasks.

Source excerpts

“A task qualified if at least two of five model families fail on it across runs. That gave 54 candidates and 3 were removed after expert review. Every Hard task passes its reference solution, fails with an empty patch, and is solved by at least one of our runs.” · Show in paper

Harness study evaluating GLM-5.3, Kimi K3, and Inkling on all 642 V2 tasks across mini-swe-agent, Pi, and OpenCode with a 1M-token context window and 50-minute budget, analyzed with a shell-command activity classifier.

Source excerpts

“GLM-5.3 (thinking on), Kimi K3 (reasoning effort max) and Inkling (xhigh) ran with a 1M-token context window and no cap on thinking tokens or turns; the only cap besides the 50-minute budget is mini-swe-agent’s default 30 s per-command timeout, which is part of that scaffold.” · Show in paper

“The activity shares of Figure 4b and the test-after-edit analysis of §4.5 come from a classifier that reads the first executable of each segment of a shell command; it matched a manual read on 147 of 150 sampled commands and on 97.2% of sampled wall-clock” · Show in paper