Headline finding
SWE-Bench Pro V2 repairs or retires 480 of the 731 original SWE-Bench Pro tasks via agentic audits, expert adjudication, blind re-implementation, and verifier hardening, resulting in a 642-task public split and a 51-task Hard split evaluated under a hardened protocol.
The task counts, defect taxonomy, and repair pipeline are consistently documented in Figure 2, Table 1, and Sections 3.1–3.5.
“Agentic audits, expert adjudication and blind re-implementation repaired or retired 480 of the 731 tasks: 23 contracted engineers spent 404 hours, each working in a language they write professionally, and every revised instruction was verified by a second engineer solving the task from the text alone. A hardened protocol restricts network access to the model endpoint and re-grades every patch in a clean container, so agents cannot retrieve upstream fixes or get credit for changes outside their patch.” · Show in paper