By Keenan Samway, Daniel D'souza, Omar Darwiche Domingues, Marzieh Fadaee, Tan Yi-Chern, Nouha DziriSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2610-17471.git
Cite
Version link
https://openversions.com/index/arxiv-2610-17471/commit/aea3480ced3fc308bf2efb9f0a029f885c0b1dd6
Version identifier (commit hash)
aea3480ced3fc308bf2efb9f0a029f885c0b1dd6

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

On ProgramBench, with the right compute allocation, a heterogeneous 17-agent team (a GPT-5.6 Sol lead with 16 Luna workers directed toward planning) comes within 1.4 points of a strong homogeneous 5-agent Sol team at 70% lower cost.

On ProgramBench, homogeneous GPT-5.6 Sol teams outperform solo agents under both minimal and heavy PIV guidance, but cost 3 to 7 times as much in inference dollars and take 1.4× to 3.8× longer in wall-clock time.

On ProgramBench, planning is the most efficient stage to delegate to workers: in a homogeneous Sol team, delegating only planning matches the reward of delegating all three stages (PIV) at half the cost, with each dollar spent on planning buying roughly 1.6× more reward than a dollar spent on implementation, verification, or the full workflow.

On ProgramBench, cheap workers (GPT-5.6 Luna) paired with a strong lead (GPT-5.6 Sol) pay off only when directed toward planning—where a Sol+Luna planning team recovers 47.5% of the all-Sol team's reward gain for 13.9% of its additional cost—whereas cheap implementers lower reward below the solo agent.

In heterogeneous teams with Luna workers on ProgramBench, the lead model determines the performance ceiling: upgrading the lead from Luna to Terra raises reward by 20.9% and from Terra to Sol by a further 15.4%, while a Terra lead with Luna workers still trails a Sol solo agent by 10.3%.

When scaling fixed-size teams of Luna workers with a Sol lead on ProgramBench from 2 to 16 workers, planning workers monotonically improve reward at every team size, whereas implementation and verification workers plateau beyond four workers.

Adding a direct send_message tool for inter-agent communication during execution does not raise reward on ProgramBench: Sol and Terra teams are unchanged within noise, and Luna teams lose 4.4–7.1 points.

The optimal workflow stage for delegation is task-dependent: unlike ProgramBench where planning is the bottleneck, on Commit0-Lite—where the repository skeleton and unit tests are provided—implementation guidance is the most effective single-stage prompt in eight of ten settings.

Methods

Hierarchical lead-worker multi-agent harness using ReAct-style agents with a standard bash tool, a delegate_agent(task, agent_id=None) tool for spawning or resuming persistent workers, and optional list_agents and wait_for_updates tools under blocking or asynchronous push scheduling.

Source excerpts

“Following the minimalist design of mini-SWE-agent (Yang et al., 2024), agents use a standard bash tool to interact with the environment and follow a ReAct-style loop (Yao et al., 2023). We adopt a top-down hierarchy where a lead agent receives the task instruction and coordinates workers through a delegate_agent(task, agent_id=None) tool.” · Show in paper

“We study two scheduling policies. Under blocking delegation, a call to delegate_agent returns only when that worker has finished. ... Under asynchronous push, a call to delegate_agent immediately returns with the agent ID and the subagent continues to work in the background.” · Show in paper

Budget-constrained team design formulation and marginal efficiency metric ηc=Rc−RsoloCc−Csolo\eta_c = \frac{R_c - R_{\text{solo}}}{C_c - C_{\text{solo}}} (reported in reward points per $100) evaluated via coordinate-wise search over workflow stage, model tier, and worker count.

Source excerpts

“We define the efficiency of a configuration c as the reward it gains per dollar of added inference cost relative to the solo agent: ηc = (Rc − Rsolo) / (Cc − Csolo)” · Show in paper

Prompting regimes comparing minimal guidance (delegation tools only) against heavy three-stage waterfall guidance (Plan-Implement-Verify, PIV) and single-stage isolation variants (P, I, V).

Source excerpts

“Each stage-specific guidance instructs the lead to complete the full workflow while using workers only in the designated phase. This design isolates the value of delegation at each phase. We then compose all three interventions into PIV, our heavy-guidance variant, to evaluate their joint effect when workers are used throughout the workflow” · Show in paper

Edit-defined phase clock that anchors each successful lead delegation call to the team's first and last detected source-file writes to partition rollouts into planning (pre-edit), implementation (active-edit), and verification (post-edit) intervals.

Source excerpts

“We anchor each delegation call to the team’s source edits. The first and last detected source writes divide a rollout into three observable intervals: time before editing, active editing, and time after editing.” · Show in paper

Benchmark evaluation on ProgramBench (200 repository-reconstruction tasks from black-box executables, 3 runs for GPT-5.6 models) and Commit0-Lite (16 Python library reimplementation tasks, 1 run).

Source excerpts

“We primarily use ProgramBench (Yang et al., 2026a), a from-scratch repository reconstruction benchmark comprised of 200 unique tasks ... Additionally, we also explore Commit0-Lite (Zhao et al., 2024) a benchmark comprised of 16 Python libraries that an agent is asked to reimplement given a repository skeleton and a set of unit tests.” · Show in paper