By Ibragim Badertdinov, Maksim Nekrashevich, Anton Shevtsov, Alexander GolubevSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2602-23866.git
Cite
Version link
https://openversions.com/index/arxiv-2602-23866/commit/41ec389476c82f0fe33747c31468e698a503a7a1
Version identifier (commit hash)
41ec389476c82f0fe33747c31468e698a503a7a1

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2602.23866v2 OpenVersions: e44f642a-a5c6-4662-8855-b7038272941e

Posted

Claims

Findings

Headline finding

SWE-rebench V2 is a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale, producing a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories with pre-built images for reproducible execution, plus 120,000+ PR-derived tasks.

Interactive setup synthesis agents consistently outperform non-interactive setup pipelines, and increasing the number of setup attempts from one to ten substantially improves repository installation success.

In interactive setup synthesis, 32k context tokens appear sufficient for most projects, while longer context can increase the risk of the agent getting stuck in loops or focusing on irrelevant details.

For automated issue clarity filtering against human-annotated SWE-bench Verified labels, the Verified+ prompt achieves the best F1 score (0.50), Verified-E achieves the highest precision (0.83), and ensembling by averaging scores yields the best overall F1 (0.43) whereas strict consensus is beneficial when precision is prioritized (0.88).

Instance-level diagnostic metadata (Code A vs. Code B* labels) separates cleanly solvable tasks from tasks with environment-, test-, or specification-induced confounding factors across evaluated frontier models.

Generating synthetic problem statements from pull request descriptions and patches for PR-derived tasks yields 77.0% clean statements (392/509) and only 2.4% explicit solution leakage (12/509) when evaluated on SWE-bench Pro pull requests.

Methods

Five-stage language-agnostic SWE task construction pipeline comprising Preliminary Data Collection, Setup Synthesis, Execution-based Validation, Filtering By Issue Clarity, and Metadata Enrichment.

Source excerpts

“The pipeline has five stages: 1. Preliminary Data Collection: Mining and filtering candidate PRs from global GitHub activity. 2. Setup Synthesis: Deploying an interactive agent to autonomously infer repository-level installation and testing scripts. 3. Execution-based Validation: Validating environments through dual-pass execution (pre- and post-fix) to extract test oracles. 4. Filtering By Issue Clarity: Removing underspecified tasks and excluding instances with potentially sensitive information. 5. Metadata Enrichment: Tagging instances with diagnostic features to enable flexible selection.” · Show in paper

Preliminary data collection from GitHub Archive and local git histories via distributed map-reduce jobs, splitting PR diffs into solution and test patches using the regex `(?i)(test(?:ing|s)?|e2e)` and applying language-tier repository star/issue thresholds.

Source excerpts

“To bypass GitHub API rate limits and enable large-scale processing, we clone repositories in distributed map-reduce jobs and extract pull request patches directly from local git histories, avoiding per-instance API requests.” · Show in paper

“For each selected pull request, we split the overall diff into a solution patch (non-test files) and a test patch (test files only). Test files are identified using the regular expression (?i)(test(?:ing|s)?|e2e).” · Show in paper

Interactive repository setup synthesis using mini-SWE-agent v1.14.4 with Qwen3-Coder-480B-A35B-Instruct on pre-built language base Docker images, plus automated repository-specific Python log parser synthesis.

Source excerpts

“We employ the mini-SWE-agent v1.14.4 (Yang et al., 2024) scaffold with Qwen3-Coder-480B-A35B-Instruct as the underlying model to generate installation and test scripts.” · Show in paper

“From a batch of raw test logs, we use Qwen3-Coder-480B-A35B-Instruct with a fixed prompt provided in Appendix A.3.3 to synthesize a repository-specific parser that maps output into a normalized schema (pass, fail, error, and test identifiers).” · Show in paper

Dual-pass execution-based validation in multi-stage Docker containers running the full project test suite pre-fix (test patch only) and post-fix (test + solution patch), repeated three times to retain only instances with stable fail-to-pass outcomes.

Source excerpts

“We then apply the test patch and run the full test suite. Next, we additionally apply the solution patch and rerun the full test suite. ... We retain only instances with at least one fail-to-pass test, ensuring a non-trivial executable signal for learning and evaluation. To reduce noise from flaky or nondeterministic tests, we rerun each candidate task three times and retain only instances whose structured test outcomes remain unchanged across runs.” · Show in paper

Automated issue clarity filtering using three independent LLM judges (gpt-oss-120b, GLM-4.7, DeepSeek-V3.2) with the Verified-E prompt, retaining instances only when all three judges rate the specification as adequate.

Source excerpts

“We score each issue with three independent LLM judges (gpt-oss-120b (OpenAI, 2025), GLM-4.7 (Team et al., 2025), and DeepSeek-V3.2 (DeepSeek-AI et al., 2025)) and retain an instance only when all three judges rate the specification as adequate for implementation.” · Show in paper

Metadata enrichment with diagnostic codes (A, B1–B7) and auxiliary interface extraction using gpt-oss-120b, plus PR-based task expansion generating synthetic problem statements from PR descriptions and patches.

Source excerpts

“The annotation is performed using the gpt-oss-120b model. In addition, for each task we automatically generate auxiliary interfaces extracted from the patches that are explicitly exercised in the test suite.” · Show in paper

“For these candidates, we generate a synthetic problem description conditioned on the pull request description and the corresponding patch, rather than relying on the raw PR text.” · Show in paper