By Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Zongwei Lv, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Lin Sun, Xiangzheng Zhang, Guangxiang Zhao, Tong YangSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2605-27922.git
Cite
Version link
https://openversions.com/index/arxiv-2605-27922/commit/4d90040d3303f34f5e2fdd05b4d85b06a9f89c1f
Version identifier (commit hash)
4d90040d3303f34f5e2fdd05b4d85b06a9f89c1f

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Harness-Bench's evaluation of seven native harnesses across ten backends on 274 general tasks and seven backends on 26 multimodal tasks reveals substantial, model-dependent performance differences where no single harness performs best across all ten backends.

Harness choice substantially affects performance within individual models, with general-task mean score gaps ranging from 5.0 to 9.5 points across nine models and reaching 39.2 points for Qwen3.8-27B, while multimodal gaps are larger for several models (e.g., Kimi-K3 ranges from 28.1 with Codex to 72.7 with OpenClaw).

Higher mean performance across models is associated with lower harness sensitivity (Spearman's ρ=−0.76\rho = -0.76 overall, or −0.67-0.67 excluding Qwen), yet full-credit attainment (R1R_1) remains sensitive even for high-scoring models.

Selecting a separate harness per model on training tasks improves held-out full-credit attainment (R1R_1) by 2.53–2.73 percentage points over a single shared harness, whereas mean-score gains remain within ±0.14\pm 0.14 points and R0.8R_{0.8} gains within 0.31 points.

Quality, completion, and resource use order harness configurations differently: OpenCode leads pooled mean score (66.3), while OpenClaw leads pooled full-credit attainment (R1=27.1%R_1 = 27.1\%) at the lowest API cost (0.1530.153 USD per task).

Additional API expenditure does not necessarily produce better delivery: with GPT-5.6 Sol, Nanobot incurs 2.97 times OpenClaw's API cost while achieving a lower full-credit rate (24.82% vs. 31.02%), and with DeepSeek-V4-Flash, Codex scores 68.34 at $0.02022 per task versus Nanobot's 60.92 at $0.03272 per task.

Correct central decisions or local checks can still produce incomplete work products, and evidence omissions can propagate through mutually consistent outputs.

Trajectory analysis of 6,864 sampled runs across all 300 tasks shows that verification is the most frequent of six recurring error categories, and that local error repair does not guarantee restored delivery unless feedback changes the next action, acquired evidence remains usable under compaction, and settled decisions are materialized into verifiable work.

Methods

Construct 300 sandboxed tasks (274 general, 26 multimodal) across six work categories and 17 subcategories with 1,408 scored modules, using LLM-free task-specific oracles that award partial credit under hard safety and risk gates (Qi=gi∑jwijxijQ_i = g_i \sum_j w_{ij} x_{ij}).

Source excerpts

“The suite contains 274 general and 26 multimodal tasks across six categories and 17 subcategories (Figure 1). ... Across the 300 tasks, 1,408 scored modules assess delivery requirements, with three to seven modules per task.” · Show in paper

“Let xij ∈ {0, 1} indicate whether module j of task i passes, with predefined weights wij ≥ 0 summing to one. The score is Qi = gi sum_j wij xij : gi = 1 when all safety and risk gates pass, and zero otherwise.” · Show in paper

Evaluate seven native harnesses across ten backends on general tasks and seven backends on multimodal tasks in isolated workspaces with a 200K-token input-context threshold, 150-turn interaction limit, temperature 1.0, and single evaluation per configuration, estimating API cost from official provider prices.

Source excerpts

“Runs use a 200K-token input-context threshold, beyond which each harness applies its default context management, and a 150-turn interaction limit. ... Each task–model–harness combination is evaluated once.” · Show in paper

“Temperature is set to 1.0, and the output-token limit is set to the maximum supported by each model. ... API expenditure is estimated from token usage using the official prices in Table 6, with the same model-specific schedule across all harnesses.” · Show in paper

Quantify model-harness sensitivity using the population SD of each model's seven harness-specific mean scores, Spearman rank correlations, and interaction residuals Imh=Qˉmh−Qˉm⋅−Qˉ⋅h+Qˉ⋅⋅I_{mh} = \bar{Q}_{mh} - \bar{Q}_{m\cdot} - \bar{Q}_{\cdot h} + \bar{Q}_{\cdot\cdot}.

Source excerpts

“We measure each model’s sensitivity as the SD of its seven harness-specific mean scores. ... We separate model-specific fit from overall strength using the interaction residual Imh = Qmh - Qm. - Q.h + Q...” · Show in paper

Annotate 6,864 score-stratified sampled trajectories using GPT-5.6 Luna with evidence-linked packets across six error categories, validated by two human annotators on 100 randomly sampled trajectories (97% inter-annotator agreement, 97–98% agreement with Luna).

Source excerpts

“The corpus contains 6,864 unique runs covering all 300 tasks. ... GPT-5.6 Luna then classifies sampled trajectories using evidence packets containing task requirements, execution events, tool feedback, and artifact excerpts.” · Show in paper

“Two human annotators independently reviewed the same 100 trajectories randomly sampled from the base samples. ... Before adjudication, complete error-label-set agreement was 97% between the human annotators and 98% and 97% between each annotator and GPT-5.6 Luna, respectively.” · Show in paper