By Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, Ludwig SchmidtSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2601-11868.git
Cite
Version link
https://openversions.com/index/arxiv-2601-11868/commit/46799bd078275431429ff05609e28c610f8485ad
Version identifier (commit hash)
46799bd078275431429ff05609e28c610f8485ad

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2601.11868v1 OpenVersions: cfc698b7-0114-4422-b473-679c6d52e255

Posted

Claims

Findings

Headline finding

Frontier models and agents resolve less than 65% of the 89 tasks in Terminal-Bench 2.0, with Codex CLI paired with GPT-5.2 achieving the highest average resolution rate of 63% (62.9% ± 3.0%), followed by Terminus 2 with Claude Opus 4.5 at 58% (57.8% ± 2.5%) and Terminus 2 with Gemini 3 Pro at 57% (56.9% ± 2.5%).

Model selection is usually more important than agent scaffold when optimizing for performance on Terminal-Bench 2.0.

There is essentially no correlation between the average number of turns (episodes) per trial and model success rates on Terminal-Bench 2.0, and higher output token count does not necessarily correlate with better performance.

State-of-the-art performance on Terminal-Bench 2.0 nearly doubled in the eight months between the release of Gemini 2.5 Pro and GPT-5.2.

Human-predicted task difficulty correlates positively with empirical difficulty (r=0.436,p<0.001r = 0.436, p < 0.001), with 93.3% of human-hard tasks also being empirically hard, whereas 54.5% of human-medium tasks are empirically hard for models.

In trajectory-level error analysis under Terminus 2, execution errors dominate for frontier closed-source models (Claude Opus 4.5 and GPT-5.2), whereas the evaluated open-source model (Qwen Coder 480B) displays a more balanced error pattern with higher errors across all failure modes.

At the command level in Terminus 2 trajectories, command error rates range from 9.2% (Grok 4) to 26.7% (GPT-OSS-120B), with command failures calling executables not installed or not in PATH being most frequent (24.1% of all failures), followed by failures when running executables (9.6%).

Methods

Terminal-Bench 2.0 task formulation and dataset construction: 89 tasks selected from 229 crowd-sourced tasks by 93 contributors, each consisting of an instruction, a Docker image, a set of unit tests, a human-written oracle solution, and a time limit, executed via the Harbor harness.

Source excerpts

“A Terminal-Bench task consists of an instruction, a Docker image, a set of tests, an example solution, and a time limit (Figure 2).” · Show in paper

“In order to design a diverse benchmark, we crowd-sourced tasks through open-source contributions, with 93 contributors creating 229 tasks. Contributors assigned expert and junior-engineer completion time estimates to their tasks. Of those 229, we selected 89 for the Terminal-Bench 2.0 dataset based on the author’s difficulty assessment and a quality assessment by three experienced human reviewers (Section 2.3).” · Show in paper

Multi-stage task verification and quality control pipeline combining automated deterministic CI checks, an LLM-backed quality check tool, an LLM-powered debug tool, an adversarial exploit agent (OpenHands), and manual review by three experienced human reviewers totaling approximately three reviewer-hours per task.

Source excerpts

“Between the three reviews, the average task received approximately three hours of combined reviewer attention, implying multiple hundreds of person-hours went into reviewing alone, excluding the time spent creating the tasks.” · Show in paper

“During task auditing, we run an adversarial exploit agent to attempt to pass the tests by cheating without actually looking at the tests and oracle solutions.” · Show in paper

Terminus 2 neutral agent scaffold: a minimal agent operating in a loop where a language model produces keystrokes executed in a tmux shell inside a Docker container using only Bash commands, with a context summarization module for long-horizon tasks.

Source excerpts

“Terminus 2 has a single tool, a headless terminal, and completes tasks using only Bash commands.” · Show in paper

“The agent operates in a simple loop where a language model produces keystrokes which are executed in a tmux shell inside a Docker container. After execution the results of the command’s output is returned to the model.” · Show in paper

Trajectory-level error analysis using a simplified three-class taxonomy (Execution, Coherence, Verification) derived from MAST, with failed Terminus 2 trials annotated via Docent and classified by GPT-5 (high-reasoning mode) as LLM-as-judge.

Source excerpts

“The result is a simplified taxonomy with three high-level classes – Execution, Coherence, and Verification —- each with subclasses.” · Show in paper

“GPT-5 (high-reasoning mode) serves as our primary judge. Compared against 120 human-labeled traces, this process achieves the closest alignment with human annotations, with an agreement of 90% (92% precision, 90% recall).” · Show in paper

Command-level error analysis using GPT-5 (medium reasoning) to identify command failures in Terminus 2 trajectories and GPT-5 (high reasoning) to classify 3,800 uniformly sampled failures into a command failure mode taxonomy.

Source excerpts

“An LLM-as-judge (GPT-5, medium reasoning, 92.4% agreement with the majority vote label as determined by three annotators reviewing 66 pairs) is used to review individual command input-output pairs from the recorded Terminus 2 trajectories and determine if a failure is observed in the output.” · Show in paper

“We use an LLM-as-judge (GPT-5, high reasoning effort, 82.0% agreement with 50 annotations provided by an author) to categorize command failures against the taxonomy. 3,800 failures are uniformly sampled and categorized across Terminal-Bench tasks and models; the results are shown in Figure 9.” · Show in paper

Benchmark integration via Adapters: translating 26 external benchmarks into Terminal-Bench's standardized schema and validating each adapter through controlled parity experiments comparing original and adapted versions.

Source excerpts

“They then conduct parity experiments—controlled comparisons where identical agents and models are evaluated on both the original and adapted versions across multiple trials. These experiments ensure that adaptation preserves the benchmark’s evaluation semantics without introducing artifacts.” · Show in paper