By Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, Brad KenstlerSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2509-16941.git
Cite
Version link
https://openversions.com/index/arxiv-2509-16941/commit/82a68e51ffa7ca1e07d049b715d3bae731a6bcb1
Version identifier (commit hash)
82a68e51ffa7ca1e07d049b715d3bae731a6bcb1

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2509.16941v2 OpenVersions: e3a09636-4b4b-46fe-b4a4-8f2afb85e355

Posted

Claims

Findings

Headline finding

SWE-BENCH PRO is a contamination-resistant benchmark of 1,865 human-verified, enterprise-level software engineering problems across 41 repositories, on which widely used coding models under a unified scaffold achieve below 45% Pass@1 performance.

SWE-BENCH PRO mitigates data contamination by exclusively selecting repositories under strong copyleft licenses (GPL) for its public (11 repositories) and held-out (12 repositories) sets, and acquiring proprietary commercial codebases from startups (18 repositories).

SWE-BENCH PRO tasks require substantial, multi-file modifications, with reference solutions spanning an average of 107.4 lines of code across 4.1 files, every problem involving at least 10 lines of change, and over 100 tasks requiring more than 100 lines of modification.

Model performance on SWE-BENCH PRO is substantially higher on the public set than on the commercial set, where the best models score less than 20% Pass@1.

Model resolve rates on SWE-BENCH PRO vary across programming languages and repositories, and degrade sharply as the number of modified files increases, with the gap between frontier and smaller open-source models widening beyond 3 files.

Removing human-authored requirements and interface specifications significantly degrades model performance on SWE-BENCH PRO because unit test verifiers become prone to false negatives.

LLM-as-a-judge failure mode analysis indicates that frontier models like Claude Opus 4.1 fail primarily on semantic understanding (wrong solutions) and syntax errors, Sonnet 4 struggles with context overflow, and smaller open-source models like Qwen3 32B exhibit high tool-use and syntax error rates.

Methods

Problems are sourced from repository commit histories by identifying pairs of consecutive commits (base and instance) to separate the diff of test-related files (test patch) from the remaining code diff (gold patch).

Source excerpts

“To collect the problems, we leverage the evolution of a codebase through its commit history. Specifically, we identify pairs of consecutive commits that together capture the resolution of an issue.” · Show in paper

“We define the test patch as the diff of test related files between the two commits. In other words, it consists of the new or modified tests introduced in the instance commit but absent in the base commit. The remaining diff, excluding the test patch, is referred to as the gold patch.” · Show in paper

Task descriptions are created through human-driven augmentation of original commits, issues, and PRs to produce a problem statement, a list of requirements grounded in the gold patch and test expectations, and optional interface specifications.

Source excerpts

“we collect and organize the available information from original sources, such as issue discussions, commit messages, or pull requests, and produce the final task description with two artifacts: (1) a problem statement, which captures the motivation for the change without extending beyond sources, and (2) a list of requirements and optionally interface, which provides the necessary details to fully understand and resolve the issue, grounded in the gold patch and test expectations when applicable.” · Show in paper

Execution environments and test suites are built and validated in three steps: manual Docker environment construction by professional software engineers, automated repeated execution of gold tests to eliminate flaky tests, and human verification of fail2pass tests to drop irrelevant or overly broad tests.

Source excerpts

“We create environments through 3 steps: First, we construct environments manually with software engineering experts. Second, we use an in-house pipeline to validate that test are not flaky and that golden tests can pass the test suite successfully. Finally, we have a human-verification of all tests in the fail2pass test list, in which irrelevant tests are dropped.” · Show in paper

Models are evaluated using the SWE-Agent scaffold with a maximum of 50 turns, the default SWE-Agent prompt, and vllm on a single node with 8 H100 GPUs for open-source models (with a $2 cost cap applied for trajectory analysis).

Source excerpts

“We use the SWE-Agent [16] scaffold. ... For open-source LLMs, we use vllm to host each model. Models are hosted on a single node, with 8 H100 Nvidia GPUs. ... Models have a maximum of 50 turns. We use the same prompt for all models, which is the default prompt from [16].” · Show in paper

“For compute constraints, analysis is done on trajectories with a max turn limit of 50 and max cost of $2. Corresponding results are in Table 5.” · Show in paper

Failure modes of unresolved agent trajectories are classified using GPT-5 as an LLM-as-a-judge prompted with the last 20 turns of each rollout and strict descriptions of hand-curated failure buckets.

Source excerpts

“We conduct an LLM-as-a-judge analysis for failure modes of different models, utilizing GPT-5 as the judge. ... For each of the models in Table 4, we programmatically filter to only unresolved instances of SWE-BENCH PRO and collect the last 20 turns of each rollout. ... With a system prompt providing strict descriptions of the failure buckets and overall SWE-Agent format, we feed the trajectory input and prompt the GPT-5 judge to first produce a 1-paragraph reasoning and then an ultimate selection of one failure mode per instance.” · Show in paper