Headline finding
Harness effectiveness in LLM-based software engineering agents depends jointly on model capability and task type: complex harnesses provide diminishing marginal gains on SWE-style issue repair as model capability improves, but can benefit stronger models on more complex and open-ended repository-level tasks.
The empirical results across SWE-bench Verified, SWE-bench Pro, ProgramBench, and GitTaskBench consistently support the joint dependence of harness effectiveness on model capability and task type.
“Experimental results show that harness effectiveness depends jointly on model capability and task type. Complex harnesses provide diminishing marginal gains on SWE-style issue repair as model capability improves, but can benefit stronger models on more complex and open-ended repository-level tasks.” · Show in paper