Headline finding
Long-Horizon-Terminal-Bench is a challenging benchmark of 46 containerized terminal tasks across nine categories with dense subtask-based grading, on which 17 frontier models average 9.8M tokens, 239 episodes, and 88.9 minutes per task, with even the strongest tested model (Grok 4.5) achieving only 28.3% pass@1 at a partial-reward threshold of 0.95 and 19.6% at 1.0 (mean pass rates of 6.4% and 3.2%, respectively).
The reported pass rates, token counts, episode counts, and execution times are directly supported by the quantitative results in Table 1, Figure 3, and Section 3.1.
“We evaluate 17 frontier models and find that agents consume on average 9.8M tokens per task, with roughly 239 episodes and 88.9 minutes of execution time per run, making Long-Horizon-Terminal-Bench substantially more demanding than prior terminal-based benchmarks. Even the strongest tested model, Grok 4.5, achieves only 28.3% pass@1 at a partial-reward threshold of 0.95 and 19.6% at a perfect-reward threshold of 1.0, while the mean pass rate across models is just 6.4% and 3.2% under the two thresholds, respectively.” · Show in paper