Search publications
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale· Method
PaperSource excerpt
“We employ the mini-SWE-agent v1.14.4 (Yang et al., 2024) scaffold with Qwen3-Coder-480B-A35B-Instruct as the underlying model to generate installation and test scripts.” · Show in paper
SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark
PaperSource excerpt
“On the 642 V2 tasks, frontier agents resolve up to 99.4% while other systems stay clearly separated, and a 51-task Hard split spreads systems from 52.9% to 98.0%.” · Show in paper
Verified
Table 2 and Figure 1 directly support the reported pass-rate ranges and separation across systems.
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
PaperSource excerpt
“Second, interactive systems consistently outperform non-interactive ones: for example, an interactive agent using Qwen3-Coder-30B-A3B-Instruct can exceed a non-interactive pipeline even when the latter is driven by Qwen3-Coder-480B-A35B-Instruct. Third, increasing the number of setup attempts improves success probability: moving from one to ten runs substantially increases the fraction of installed repositories, in some settings approaching a twofold improvement.” · Show in paper
Verified
Table 2 directly confirms both that the 30B interactive agent outperforms the 480B non-interactive baseline at all pass@k values and that pass@10 is roughly double or more than pass@1 across interactive configurations.
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
PaperSource excerpt
“We introduce SWE-rebench V2, a language-agnostic automated pipeline for harvesting executable real-world SWE tasks and constructing RL training environments at scale. ... Using this pipeline, we construct a dataset of 32,079 tasks spanning 20 languages and 3,617 repositories, with pre-built images for reproducible execution. To further scale training data, we additionally release 120,000+ tasks with installation instructions, fail-to-pass tests and rich metadata, where the problem statement is generated based on the original pull request description.” · Show in paper
Verified
The pipeline design, stage-by-stage yield counts in Table 1, and dataset statistics directly support the claimed scale and language coverage.
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
PaperCodeDataSource excerpt
“To address these limitations, we introduce a novel, automated, and scalable pipeline to continuously extract real-world interactive SWE tasks from diverse GitHub repositories. Using this pipeline, we construct SWE-rebench, a public dataset comprising over 21,000 interactive Python-based SWE tasks, suitable for reinforcement learning of SWE agents at scale.” · Show in paper
Verified
The manuscript thoroughly documents the four-stage automated pipeline, provides concrete funnel metrics, and reports the exact counts of 21,336 tasks (3,468 repos) for the dataset and 294 tasks (169 repos) for the benchmark subset.
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
PaperSource excerpt
“To quantify leakage in PR-derived tasks, we evaluated 509 PR-derived tasks constructed from SWE-bench Pro pull requests. An LLM judge compared each generated statement against the reference problem statement, requirements, and interfaces, and rated whether the statement introduced implementation details beyond the reference. Under this protocol, 392/509 tasks (77.0%) were clean, 117/509 (23.0%) contained some leakage, and only 12/509 (2.4%) contained explicit solution leakage.” · Show in paper
Verified
The reported counts (392/509 clean, 117/509 some leakage, 12/509 explicit leakage) directly support the claim under the stated LLM-judge protocol.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“Managed coding assistants, Claude Code (Opus 4.6) and Codex (GPT-5.4), agree on the primary defect category, including no defect, at κ = 0.80 over all 89 TB2.0 tasks. Lower-cost API auditors agree on the binary triage decision (flagged vs. clean) over all 89 tasks (Table 1), but less on attribution: per-category agreement (one-vs-rest, since a task can carry multiple labels) is moderate for instruction–verification mismatch (κ = 0.60) and environment issues (κ = 0.59), and low for task drift (κ = 0.21–0.41).” · Show in paper
Verified
Table 1 and Section A.2 support the reported kappa values for both managed coding assistants and lower-cost API auditors.
Scaling Inference-Time Compute with Teams of Coding Agents· Method
PaperSource excerpt
“We primarily use ProgramBench (Yang et al., 2026a), a from-scratch repository reconstruction benchmark comprised of 200 unique tasks ... Additionally, we also explore Commit0-Lite (Zhao et al., 2024) a benchmark comprised of 16 Python libraries that an agent is asked to reimplement given a repository skeleton and a set of unit tests.” · Show in paper
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“As shown in Table 8, compared to an average gold patch, model generated patch files that apply correctly are less than half the total length (74.5 versus 30.1 lines) of gold edit patch files, and rarely edit more than a single file.” · Show in paper
Verified
Table 8 confirms that all evaluated models' applied patches average at most 30.1 total lines (less than half of All Gold's 74.5 lines) and 1.0-1.1 files edited.
Bursts That Outlast Themselves: Persistent Degradation in Prefix-Cached Agent Serving· Method
PaperSource excerpt
“We replay whole-session shapes—initial context, per-turn output, per-turn context growth and inter-turn gaps—of 200 trajectories sampled from public mini-SWE-agent leaderboard runs on SWE-bench Verified by six 2026 models” · Show in paper
End-to-End Context Compression at Scale
PaperCodeSource excerpt
“In Figure 13, LCLMs achieve the highest accuracy across all compression ratios on GSM8K, with particularly strong gains over baselines at higher compression ratios.” · Show in paper
Verified
Table 9 and Figure 13 directly support the claim across all three compression ratios.
Measuring AI Ability to Complete Long Software Tasks
PaperSource excerpt
“Controlling for task length, we find models perform worse on tasks that have higher messiness scores. Notably, trends in AI agent performance over time are similar for the lower and higher messiness subsets of these tasks (see Figure 12). In particular, we find no evidence of plateaus in performance trends specific to the higher messiness subset.” · Show in paper
Verified
The regression results and split trend plots in Figures 12 and 13 support the claim.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“Claude's initial C++ failures are overwhelmingly due to incorrect system-library assumptions (SLAR of 87.5%), and every one of those failures is eventually repaired after feedback. Codex shows a lower but still meaningful assumption rate (17.9%), again with perfect recovery once the missing package is revealed. Gemini's C++ failures arise from other causes rather than missing system libraries.” · Show in paper
Verified
Table 6 and Section 6.4 directly substantiate the SLAR and EGAR figures for all three agents.
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents· Method
PaperCodeDataSource excerpt
“All evaluations on SWE-rebench are conducted by our team by using a fixed scaffolding, i.e., every model is assessed by using the same minimal ReAct-style agentic framework [Yao et al., 2023], identical prompts and default generation hyperparameters as recommended by model developers. We standardize the context length to 128K tokens for all evaluations, unless a model only supports a shorter context.” · Show in paper
End-to-End Context Compression at Scale· Method
PaperCodeSource excerpt
“We segment the input into fixed-size chunks of 1,024 encoder tokens, compress each chunk, and assign it a segment identifier. The model receives the entire compressed sequence in a single prompt, together with one expand tool. At each turn, the model can issue a native tool call to expand with a segment identifier (Figure 6), and the tool returns the original text of that segment.” · Show in paper
QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
PaperCodeSource excerpt
“Finding 4: With the same training-token budget, QAD on fewer 32K sequences generalizes better than QAD on more 4K sequences, improving average accuracy across AIME25, BigCodeBench, and LiveCodeBench-v6 by 1.9 percentage points. QAD also generalizes across domains, with the strongest transfer observed from code training to mathematical reasoning.” · Show in paper
Verified
The token-matched experiments in Table 7 and analysis on page 20 substantiate both the 1.9-point gain and cross-domain transfer.
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“This paper delivers cheap, precise, verifiable control of a coding agent’s decisions by LLM-written weight edits, and a measured account of when such control does and does not raise solve rate.” · Show in paper
Verified
The manuscript thoroughly substantiates both the precise, dose-dependent decision control and the nuanced, negative/unconfirmed solve-rate findings with appropriate controls and pre-registered tests.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“The unanimous consensus rate is 0% in every language, meaning Claude never converged to the same dependency declaration across all three trials for any task in any ecosystem. Intra-agent Jaccard scores are low throughout, with JavaScript again the worst case.” · Show in paper
Verified
Table 5 provides complete data from the 400 stochastic trials of Claude Code confirming 0% UCR and low intra-agent Jaccard scores across all four languages.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“In this setting, we find that models generally perform worse at this task than when generating patch files; for instance, Claude 2 scores at 2.2% compared to 4.8% in the main table for “oracle” retrieval. Even when controlling for instance length, generating on the shorter half of the task instances by input tokens yields 3.9% compared to 7.8% for generating patches with Claude 2.” · Show in paper
Verified
The reported comparison for Claude 2 (2.2% vs. 4.8% overall, and 3.9% vs. 7.8% on shorter instances) supports the claim.
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
PaperSource excerpt
“Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman ρ = 0.958). Beyond accuracy, we evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock and output-token budgets can materially change apparent agent performance” · Show in paper
Verified
All components of the headline claim are directly supported by the reported audit counts (28/89 tasks), the 19-agent evaluation (Spearman rho = 0.958, rank shifts up to 3), and the 32x timeout sweep results.