By Haichuan Hu, Quanjun Zhang, Shengcheng Yu, Zhifei Chen, Tianyu Luo, Chunrong Fang, Zhenyu Chen, Liang XiaoSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2609-32459.git
Cite
Version link
https://openversions.com/index/arxiv-2609-32459/commit/3493b7a4186df9cd099feaf5a26813872399f173
Version identifier (commit hash)
3493b7a4186df9cd099feaf5a26813872399f173

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2609.32459v1 OpenVersions: 5dee244a-a19d-44df-b25f-0ff4f83a9624

Posted

Claims

Findings

Headline finding

Harness effectiveness in LLM-based software engineering agents depends jointly on model capability and task type: complex harnesses provide diminishing marginal gains on SWE-style issue repair as model capability improves, but can benefit stronger models on more complex and open-ended repository-level tasks.

On SWE-bench Verified and SWE-bench Pro, the marginal benefit of complex harnesses over mini-SWE-agent generally decreases as model capability improves, though the convergence pattern differs across model families (smooth narrowing for Qwen, non-monotonic for DeepSeek).

On newer and more complex SE tasks such as ProgramBench and GitTaskBench, complex harnesses like OpenCode provide greater benefits to stronger models (e.g., Qwen3.7-Max and DeepSeek-V4-Pro) rather than weaker models.

Individual harness components have heterogeneous effects on ProgramBench: tool registry and task-specific subagents provide the most stable and substantial gains, planning and lazy skills offer smaller or model-dependent improvements, while context compression and general subagents degrade repository-generation performance.

Combining all five representative components in NanoHarness achieves the strongest performance among variants on ProgramBench, improving over mini-SWE-agent by 7.37 percentage points on Qwen3.7-Max and 6.21 percentage points on DeepSeek-V4-Pro, and recovering most of the gains of product-level harnesses (OpenCode and Claude Code).

Effective harness mechanisms improve SE-agent performance through more structured interaction with controlled prompt growth and by regularizing probing behavior—reducing both excessive low-value probes and insufficient shallow validation on ProgramBench.

Methods

Joint model-harness-task experimental matrix evaluating 10 open-weight models (5 Qwen, 5 DeepSeek) ranked by the Coding Index under two contrasting harnesses (mini-SWE-agent and OpenCode) across three SE benchmarks (SWE-bench Pro, ProgramBench, GitTaskBench) with unified execution limits (temperature 0.0, 250/300 step limits, 6-hour wall-clock limit, 3-minute command timeout).

Source excerpts

“By combining these factors in a unified three-dimensional experimental matrix, we evaluate each model under each harness across each task, allowing us to compare their individual effects and analyze how they interact in LLM-based SE agents.” · Show in paper

“We use a unified execution protocol across all experiments. For model inference, we set the temperature to 0.0 for all models to reduce randomness. For SWE-bench Pro and GitTaskBench, we follow the original evaluation settings and limit each task to 250 agent steps. For ProgramBench, we reduce the step limit from the original 1,000 steps to 300 steps to reduce evaluation cost.” · Show in paper

Modular component-level ablation using NanoHarness, built on top of mini-SWE-agent with five independently toggleable mechanisms (tool registry, context compression, explicit planning, general/task-specific subagents, and lazy skills), evaluated individually and in combination on the full 200-instance ProgramBench set.

Source excerpts

“To study the mechanisms behind representative components, we implement a modular harness called NanoHarness on top of mini-SWE-agent, where each mechanism can be enabled or disabled independently. Specifically, we instantiate five representative mechanisms: tool registry, context compression, explicit planning, subagents, and lazy skills.” · Show in paper

“The component-level experiments in RQ2 and RQ3 use the full 200-instance ProgramBench set.” · Show in paper

Trajectory-level behavioral and failure mode analysis measuring sample-level agent steps, tool calls, prompt/response tokens, and probing-count distributions across NanoHarness configurations, supplemented by manual inspection of failed trajectories and case studies.

Source excerpts

“Specifically, RQ3.1 analyzes the differences in agent steps, tool calls, and token usage across different configurations of NanoHarness, while RQ3.2 measures the distribution patterns of probing behavior and uses case studies to illustrate shifts in failure modes.” · Show in paper

“Through manual inspection of failed and low-scoring trajectories produced by mini-SWE-agent, we categorize failures by probing count and validation behavior” · Show in paper