By Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, Boris YangelSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2505-20411.git
Cite
Version link
https://openversions.com/index/arxiv-2505-20411/commit/2b50a56a8bfdce5bf10200e22f2b6833e539c550
Version identifier (commit hash)
2b50a56a8bfdce5bf10200e22f2b6833e539c550

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

The authors introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories, constructing SWE-rebench, a public dataset of 21,336 verifiable Python-based SWE tasks from 3,468 distinct repositories, as well as a continuously updated, decontaminated benchmark of 294 tasks from 169 repositories.

The agentless LLM-driven installation recipe generation approach (using Qwen2.5-72B-Instruct with up to three candidates and iterative refinement) successfully produces a working installation recipe for at least one task in 31% of all repositories.

Fine-tuning Qwen 2.5-72B-Instruct on SWE-bench Verified human annotations achieves 81% accuracy (weighted F1: 0.82) on Task Complexity (improving over the 68% baseline), 67% accuracy (weighted F1: 0.65) on Test Patch Correctness, and 79% accuracy (weighted F1: 0.76) on Issue Clarity.

Comparing model performance on SWE-rebench (March–April 2025) against SWE-bench Verified suggests that performance of some language models on SWE-bench Verified might be inflated due to contamination issues.

GPT-4.1 is the only evaluated model whose performance noticeably declined on the March–April 2025 subset (26.7% resolved) compared to the January 2025 subset (31.1% resolved).

Qwen3 models perform similarly with or without think mode enabled, and in some cases the no-think variant slightly surpasses the think version, with nearly identical pass@5 scores.

Qwen2.5-Coder-32B-Instruct underperforms expectations on SWE-rebench (2.7%–3.2% resolved) due to instruction-following problems, frequently hallucinating environment responses or entering loops of formatting errors.

Methods

Preliminary task collection downloads ~450,000 pull requests linked to issues created before May 1, 2025 from >30,000 permissively licensed Python repositories (>75% Python code) via GitHub Archive and local GitHub clones, filtering for resolved issues, merged PRs into main, single-issue links, >10-char descriptions, test modifications plus non-test code changes, and 1–15 modified files.

Source excerpts

“To initiate the dataset building process, we download approximately 450,000 pull requests linked to issues created before May 1, 2025. These originate from over 30,000 repositories that feature permissive licenses granting broad usage rights (see Appendix D for the list of included license types) and where Python constitutes over 75% of the codebase lines of code.” · Show in paper

Automated installation instruction configuration groups task instances by normalized major.minor git tag versions, selects the most recent base_commit per group, and uses Qwen2.5-72B-Instruct in an agentless two-step process (file identification and JSON recipe extraction) with up to three candidate recipes and iterative error-log refinement.

Source excerpts

“We use the Qwen2.5-72B-Instruct model [Qwen et al., 2025] (prompt in Appendix B.2) to generate up to three candidate JSON recipes per task. If an error occurs during the subsequent scripted installation or test execution (derived from a recipe), the LLM attempts to refine that recipe by analyzing error logs and the original instructions (see correction prompt in Appendix B.4).” · Show in paper

Execution-based installation verification builds isolated containers using buildah and tmpfs on TractoAI, verifies that at least one test-patch test fails before and passes after the solution patch (while passing tests remain passing), and freezes exact dependency versions via pip freeze and conda env export.

Source excerpts

“Install project dependencies in an isolated container using buildah. We utilize tmpfs for file system operations to minimize disk I/O and accelerate builds. Execute tests and parse logs to identify tests validating the solution. Build final container images upon successful verification. Record exact dependency versions (via pip freeze, conda env export) after a successful setup to mitigate reproducibility issues from unpinned dependencies in Python projects, ensuring consistent environment recreation in the future.” · Show in paper

Automated instance quality assessment fine-tunes Qwen 2.5-72B-Instruct on over 3,800 human-annotated examples from SWE-bench Verified (75/25 train/validation split) to independently predict binary labels for Issue Clarity, Task Complexity, and Test Patch Correctness.

Source excerpts

“We fine-tune Qwen 2.5-72B-Instruct using annotations from SWE-bench Verified. For each of the over 3,800 examples, the model receives the issue description, the canonical solution patch, and the test patch as input. It is then prompted to predict one of three binary quality labels: Issue Clarity, Task Complexity, or Test Patch Correctness.” · Show in paper

Centralized and standardized benchmark evaluation runs every model five times on 294 SWE-rebench tasks using a fixed minimal ReAct-style scaffolding, 128K context length, textual command generation (no function calling), and developer-recommended generation hyperparameters, reporting mean resolved rate, SEM, and pass@5.

Source excerpts

“All evaluations on SWE-rebench are conducted by our team by using a fixed scaffolding, i.e., every model is assessed by using the same minimal ReAct-style agentic framework [Yao et al., 2023], identical prompts and default generation hyperparameters as recommended by model developers. We standardize the context length to 128K tokens for all evaluations, unless a model only supports a shorter context.” · Show in paper

“To capture performance variability, we run each model five times on the full benchmark. We additionally report both the standard error of the mean (SEM) and pass@5 metrics to provide a statistically grounded and more reliable assessment of each model performance.” · Show in paper