By Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence ChanSourcearXiv
Fork
Clone
git clone https://openversions.com/git/index/arxiv-2503-14499.git
Cite
Version link
https://openversions.com/index/arxiv-2503-14499/commit/b701b839a54f06a5d8c4aafe5944deac6b452d8f
Version identifier (commit hash)
b701b839a54f06a5d8c4aafe5944deac6b452d8f

Fork to create your own copy with credit to the source. Clone to get a Git copy on your computer.

Latest version

Checks complete

Archive from arXiv Source: https://arxiv.org/abs/2503.14499v4 OpenVersions: 6b372501-e615-4a24-9f78-f4c74819fff4

Posted

Claims

Findings

Headline finding

The 50%-task-completion time horizon of frontier AI models on the combined HCAST, RE-Bench, and SWAA task suite has grown exponentially from 2019 to early 2025, doubling approximately every 7 months (207 days, 95% CI 166-240 days).

On the combined task suite, GPT-2 has a 50% time horizon of 2 seconds while o3 has a 110-minute time horizon and lies above the long-run 2019-2025 trend (p = 0.006), which may imply the trend in 2024 and early 2025 is faster.

The 80%-task-completion time horizon exhibits a similar doubling time (204 days) to the 50% time horizon (207 days), within margin of error, though models' 80% time horizons are 4-6x shorter.

Naively extrapolating the 2019-2025 time horizon trend implies that AI will reach a >1 month (167 work hours) 50% time horizon between mid-2028 and mid-2031, though extrapolation is affected by external validity concerns and future trend changes.

Model 50% time horizons computed from SWE-bench Verified tasks follow an exponential trend from late 2023 through 2024 with a doubling time of around 70 days, which is shorter than the 143-day doubling time on HCAST + SWAA + RE-Bench for 2024 models.

Controlling for task length, AI agents perform worse on HCAST tasks with higher messiness scores (b = -0.081, R^2 = 0.251), but trends in AI agent performance over time are similar for lower and higher messiness subsets, with no evidence of plateaus specific to the higher messiness subset.

On five uncontaminated internal repository pull requests, contract baseliners take 5-18x longer to resolve issues than repository maintainers, and AI agent performance is worse than predicted by maintainer time-to-complete but consistent with contractor time-to-complete.

Increases in AI models' time horizons appear to be driven by improved tool use capabilities, greater ability to adapt to mistakes, and improved logical reasoning and code generation.

Methods

Construct a 170-task evaluation suite combining 97 HCAST tasks (1 minute to 30 hours), 7 RE-Bench ML research engineering tasks (8 hours), and 66 newly created single-step Software Atomic Actions (SWAA) tasks (1 to 30 seconds).

Source excerpts

“Our tasks comprise three distinct task suites: 1. A subset of HCAST [8]: 97 diverse software tasks ranging from 1 minute to 30 hours. 2. RE-Bench [2]: 7 difficult ML research engineering tasks, all eight hours long. 3. Software atomic actions (SWAA): 66 single-step tasks representing short segments of work by software developers, ranging from 1 second to 30 seconds.” · Show in paper

Estimate task difficulty ratings using over 800 human baselines from skilled professionals, taking the geometric mean time of successful human baseline runs t_task.

Source excerpts

“In total, we use over 800 baselines totaling 2,529 hours, of which 558 baselines (286 successful) come from HCAST and RE-Bench, and 249 (236 successful) from the shorter SWAA tasks.” · Show in paper

“where t_task is the geometric mean time of successful human baselines, and h_agent and β_agent are learned parameters, with h_agent representing the 50% time horizon.” · Show in paper

Compute each agent's 50% time horizon h_agent by binarizing continuous task scores via task-specific human performance thresholds, weighting tasks by 1/sqrt(n) for family size n, and fitting the logistic regression p_success(agent, task) = σ((log h_agent - log t_task) · β_agent).

Source excerpts

“Once we have agent success rates and time ratings for each task, we fit time horizons by performing the following logistic regression: p_success(agent,task) = σ((log h_agent − log t_task) · β_agent)” · Show in paper

“To fit equation 1, we weight tasks by diversity, such that a task from a family of size n gets weight 1/√n.” · Show in paper

Estimate the temporal trend in time horizon by Ordinary Least Squares regression of log(model_horizon) = α + β · release_date, with 95% confidence intervals computed from 10,000 samples of a three-level hierarchical bootstrap over task families, tasks, and runs.

Source excerpts

“In addition, we linearly regress log(time horizon) against release date, finding that time horizon has doubled every 207 days with a 95% bootstrapped confidence interval 166-240 days (roughly ±19%). Error bars are calculated via 10,000 samples from a three-level hierarchical bootstrap over task families, then tasks, then runs.” · Show in paper

Quantify task 'messiness' by labeling HCAST and RE-Bench tasks on the presence or absence of 16 messiness factors (summed to a 0-16 score) and regressing excess success rates against messiness.

Source excerpts

“We labeled RE-bench and HCAST tasks on the presence or absence of these 16 messiness factors, then summed these to obtain a “messiness score” ranging from 0 to 16.” · Show in paper