By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Chengsong Huang, Ergan Shang, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, Leowei LiangSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2607-08964.git
Cite
Version link
https://openversions.com/index/arxiv-2607-08964/commit/c1e3d08aa2789364ac29d817e652e5c366741967
Version identifier (commit hash)
c1e3d08aa2789364ac29d817e652e5c366741967

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Claims

Findings

Headline finding

Long-Horizon-Terminal-Bench is a challenging benchmark of 46 containerized terminal tasks across nine categories with dense subtask-based grading, on which 17 frontier models average 9.8M tokens, 239 episodes, and 88.9 minutes per task, with even the strongest tested model (Grok 4.5) achieving only 28.3% pass@1 at a partial-reward threshold of 0.95 and 19.6% at 1.0 (mean pass rates of 6.4% and 3.2%, respectively).

Dense, subtask-level grading is necessary to rank models and capture partial progress on Long-Horizon-Terminal-Bench, whereas binary pass/fail evaluation collapses most models into tie groups and hides substantial progress.

Under binary pass/fail evaluation at R >= 0.95, models tied at one solved task span mean rewards from 0.27 (GPT-5.4) to 0.32 (GLM 5.2), and pass rate and mean reward are positively but only moderately correlated (Spearman ρ=0.74\rho = 0.74).

Dense grading reveals near-misses (0.75≤R<0.950.75 \le R < 0.95) that occur nearly twice as often as passes (90 vs. 50) across all 17×4617 \times 46 model-task runs.

Higher inference spending alone is insufficient to guarantee better long-horizon performance on Long-Horizon-Terminal-Bench, as average per-task costs span from about $3.6 to $26 and the most expensive models (GPT-5.4 at $25.77/task, GPT-5.6-sol at $21.14/task, GPT-5.5 at $21.00/task) achieve lower pass rates than Grok 4.5 ($10.95/task, 28.3% pass@1 at R≥0.95R \ge 0.95).

The main bottleneck for current frontier agents on Long-Horizon-Terminal-Bench is not local execution correctness, but reliable long-horizon completion within budget, as 79% of unresolved runs (518/660) end in 90-minute timeouts with mean rewards ranging from 0.10 to 0.35.

Voluntary early exits (19% of unresolved runs, 124 runs) reveal weak self-verification and overconfidence, including 14 false-finish runs where agents exit with R≥0.75R \ge 0.75 and substantial time remaining despite not satisfying the hidden verifier.

Methods

Each Long-Horizon-Terminal-Bench task is specified in the Harbor format as a containerized Docker terminal environment containing all relevant assets, code, data, tools, and helper scripts, paired with a natural-language instruction, a task configuration file, and an oracle implementation or simulator for grading.

Source excerpts

“A Long-Horizon-Terminal-Bench task follows the Terminal-Bench formulation [22]: it is specified as a Harbor task with (1) a natural-language instruction, (2) a Docker image that defines the terminal environment, (3) a task configuration file, and (4) an oracle implementation or simulator used for grading.” · Show in paper

Tasks are graded by a deterministic in-container grader that computes normalized scores rk∈[0,1]r_k \in [0, 1] for KK semantically meaningful subtasks (binary, continuous/thresholded, or episode-aggregating) and combines them into an overall task reward R=∑k=1Kwkrk∑k=1KwkR = \frac{\sum_{k=1}^K w_k r_k}{\sum_{k=1}^K w_k}.

Source excerpts

“Each Long-Horizon-Terminal-Bench task is graded by a deterministic grader that runs inside the container at the end of a rollout. Instead of returning a strict single pass/fail, the grader computes scores for a small set of subtasks {s1, . . . , sK} that together describe the intended workflow. For every subtask sk it outputs a normalized score rk \in [0, 1]; the task reward is R = \sum_{k=1}^K w_k r_k / \sum_{k=1}^K w_k, where w_k are non-negative weights (by default all equal, with higher weight on the final goal when needed).” · Show in paper

“- Binary subtasks. ... - Continuous or thresholded subtasks. ... - Episode-aggregating subtasks.” · Show in paper

Each benchmark problem is constructed as a deliberately broken terminal-only project with reproducible public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier with dynamic stress cases, calibrated using Deepseek-V4-Pro under 1.5-hour budgets to filter 120 candidates down to 46 tasks.

Source excerpts

“For each problem, we build a complete but deliberately broken terminal-only project. ... Each task includes reproducible public asset-generation scripts, a weak baseline implementation, an official gold solution, a multi-step solve.sh, and a hidden verifier.” · Show in paper

“We calibrate difficulty by repeatedly running Deepseek-V4-Pro [4] under 1.5-hour time budgets and adjusting task design until the tasks are challenging but still solvable in principle. We generated 120 candidate tasks with quality filtering, where the final benchmark contains 46 tasks implemented in the Harbor format [22], all with containerized environments and a shared agent harness.” · Show in paper

Seventeen frontier models are evaluated using the Harbor framework with the Terminus-2 agent harness (and Codex for GPT-5.3) under a 90-minute timeout, measuring pass@1 at R≥0.95R \ge 0.95 (and 0.9,1.00.9, 1.0), mean normalized reward, episodes per task, wall-clock time, token usage, and estimated USD cost.

Source excerpts

“We evaluate models using the Harbor framework with the Terminus-2 agent harness [10], which interacts with each task environment through a single long-horizon terminal session. Additionally, for GPT-5.3, we instead use Codex [28] as the agent harness.” · Show in paper

“For each model we report pass@1 (fraction of tasks whose normalized reward is at least 0.95), mean normalized reward across all tasks, number of episodes to complete the task, length of time, and estimated dollar cost and token usage per task.” · Show in paper