Search publications
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
PaperCodeSource excerpt
“Individual harness components have heterogeneous effects. Tool registry and task-specific subagents provide the most stable and substantial gains, planning and lazy skills offer smaller or model-dependent improvements, while context compression and general subagents can hurt performance on repository-generation tasks.” · Show in paper
Verified
The ablation results on the full 200-instance ProgramBench set for both models directly substantiate the stated directional and relative effects of all five components.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“On these standard-library-sufficient tasks, Claude introduced external packages in 91.8% of these tasks on macro average, reaching 100% in C++ and remaining at or above 84% in every other ecosystem. Gemini is similarly aggressive in JavaScript. Codex is much more conservative in Python and JavaScript, though it still adds unnecessary libraries in more than half of eligible C++ tasks.” · Show in paper
Verified
The UDR analysis in Section 6.5 supports the claim, though the paper notes in Section 6.6 that standard-library-sufficient task annotations are fixed manual annotations.
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency.” · Show in paper
Verified
The empirical results across 1,000 runs substantiate that all three tested agents frequently over-specify, under-specify, and inconsistently declare dependencies even when they ultimately pass execution.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“In Table 7 we show model results in the “oracle” retrieval setting, partitioned by date, for PRs created before or after 2023. We find that for most models there’s little difference in performance before or after this date, with the exception of GPT-4. We consider this result to be largely promising as it suggests that despite models having been exposed to some version of an repository’s codebase, they are unlikely to “cheat” to address issues simply by generating a more recent version of the repository.” · Show in paper
Verified
Tables 7 and 21 support the claim that resolve rates do not consistently favor older issues.
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
PaperCodeSource excerpt
“Experimental results show that harness effectiveness depends jointly on model capability and task type. Complex harnesses provide diminishing marginal gains on SWE-style issue repair as model capability improves, but can benefit stronger models on more complex and open-ended repository-level tasks.” · Show in paper
Verified
The empirical results across SWE-bench Verified, SWE-bench Pro, ProgramBench, and GitTaskBench consistently support the joint dependence of harness effectiveness on model capability and task type.
RULER: What's the Real Context Size of Your Long-Context Language Models?· Method
PaperCodeSource excerpt
“We evaluate eight open-sourced models at various context sizes across 18 task configurations. Each task can then be represented with a vector of model performance at various context sizes. The 18 task vectors are then clustered via agglomorative clustering algorithm, using correlation coefficient as the distance metric.” · Show in paper
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software
PaperSource excerpt
“Python is the most convergent ecosystem, yet even there mean pairwise Jaccard similarity remains below 0.5 across all agent pairs, indicating that fewer than half of declared packages are shared on average. JavaScript is the most divergent, with Codex and Gemini agreeing on as little as \bar{J} = 0.073, meaning the two agents share fewer than one in thirteen declared packages for the same task. The complete disjointness rate is highest in JavaScript (63.3%), followed by C++ (57.7%) and Java (56.2%), with Python lowest at 36.0%, yielding a macro-average CDR of 53.3% across the four ecosystems.” · Show in paper
Verified
Figure 9 and Section 6.3 directly support the low cross-agent Jaccard similarities and 53.3% macro-average complete disjointness rate.
Scaling Inference-Time Compute with Teams of Coding Agents
PaperSource excerpt
“Implementation guidance is the most effective single-stage prompt in eight of ten settings, while planning, the most efficient stage on ProgramBench, is never clearly best.” · Show in paper
Verified
Figure 30 supports the 8-of-10 observation, and the authors appropriately qualify these differences as directional due to the small 16-task sample size and single-run evaluation.
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
PaperSource excerpt
“As shown in Figure 5, we see that as total context length increases, Claude 2’s performance drops considerably; behavior that is also observed in other models. In our evaluation settings, models see a lot of code that may not be directly related to solving the issue at hand, and they seem to frequently struggle with localizing problematic code needing to be updated.” · Show in paper
Verified
Figure 5 and Tables 2-3 directly demonstrate that longer input contexts degrade model resolve rates even when recall of relevant files increases.
Measuring AI Ability to Complete Long Software Tasks
PaperSource excerpt
“We find large differences in the speed at which different human groups complete the internal PR tasks, with contractors taking 5-18x longer to fix issues than repo maintainers. We also find that AI agent performance is worse than would be predicted by maintainer time-to-complete as a measure of task length, but is consistent with contractor time-to-complete,” · Show in paper
Verified
Table 5 documents the 5.5x-18.6x slowdown and Table 4 documents the model scores.
Measuring AI Ability to Complete Long Software Tasks· Method
PaperSource excerpt
“We labeled RE-bench and HCAST tasks on the presence or absence of these 16 messiness factors, then summed these to obtain a “messiness score” ranging from 0 to 16.” · Show in paper
Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software· Method
PaperSource excerpt
“Every agent a \in \mathcal{A} receives every prompt \pi_T for every task-language pair (T, l) \in \mathcal{T} \times \mathcal{L}, yielding |\mathcal{T}| \times |\mathcal{L}| \times |\mathcal{A}| = 50 \times 4 \times 3 = 600 primary instances. To assess the stochastic variability of the Gen and Repair skills, a_1 is additionally executed for two further independent trials per task-language pair, each initialized from a fresh conversation with no memory of prior attempts, contributing 50 \times 4 \times 2 = 400 stochastic trials. ... The total study scale is thus 1000 evaluation runs (Table 2).” · Show in paper
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?· Method
PaperSource excerpt
“Therefore, we choose to use BM25 retrieval (Robertson et al., 2009) to retrieve relevant files to provide as context for each task instance. We experiment with three different maximum context limits, and simply retrieve as many files as fits within the specified limit. ... For analysis purposes we also consider a setting where we “retrieve” the files edited by the reference patch that solved the issue on GitHub.” · Show in paper
SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale
PaperSource excerpt
“First, while long context can help on some repositories, 32k tokens appear sufficient for most projects. Trivial setups are resolved quickly regardless of context, and longer context can increase the risk of the agent getting stuck in loops or focusing on irrelevant details.” · Show in paper
Verified
The pass@1 and pass@10 results in Table 2 support that 32k tokens capture most of the achievable performance.
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents· Method
PaperCodeSource excerpt
“To study the mechanisms behind representative components, we implement a modular harness called NanoHarness on top of mini-SWE-agent, where each mechanism can be enabled or disabled independently. Specifically, we instantiate five representative mechanisms: tool registry, context compression, explicit planning, subagents, and lazy skills.” · Show in paper
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
PaperCodeSource excerpt
“Models maintain relatively stable performance for single-file problems but exhibit sharp declines as file count increases. Notably, the performance gap between frontier and smaller models widens dramatically beyond 3 files, with Claude Opus 4.1 and OpenAI GPT-5 maintaining above 10% resolve rates even for problems involving 10+ files, while open-source alternatives approach near-zero performance.” · Show in paper
Verified
Figure 3 visually confirms the higher resolve rates in Go/Python, variance across repositories, and the sharp decline in resolve rate as file count increases beyond 3 files.
Beyond the Model: Demystifying Harness Effects in Software Engineering Agents
PaperCodeSource excerpt
“Harness mechanisms affect failure modes by regularizing probing behavior. On ProgramBench, effective components such as task-specific subagents reduce both excessive low-value probes and insufficient shallow validation. By combining these components, NanoHarness enables more stable behavioral exploration and helps avoid wasteful probing and premature implementation.” · Show in paper
Verified
Table 7, Figure 13, and the two concrete trajectory case studies support the claim that NanoHarness regularizes exploration with controlled prompt growth.
Programming a Coding Agent's Weights by Hand
PaperSource excerpt
“Claim tested: control of a decision is enough to solve more tasks. It is not, for every direction other than explore-more.” · Show in paper
Verified
The per-arm solved counts in Section 4.6 and Appendix G directly support this negative finding.
SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents
PaperCodeDataSource excerpt
“Our approach successfully produces a working installation recipe for at least one task in 31% of all repositories.” · Show in paper
Verified
The 31% repository-level success rate is directly reported as an empirical outcome of running the pipeline, consistent with the ~14% task-level acceptance rate in Table 6.
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
PaperSource excerpt
“Codex CLI resolution rate increases by 52% when using GPT-5.2 instead of GPT-5-Nano, while Gemini-2.5-Pro sees a 17% increase in resolution rate when paired with Terminus 2 instead of OpenHands, implying that model selection is usually more important than agent scaffold when optimizing for performance.” · Show in paper
Verified
The numbers in Table 2 match the cited differences and support the qualitative conclusion that model choice generally has a larger impact than scaffold choice.