Task-Completion Time Horizons of Frontier AI Models

Research question

How does the difficulty of software tasks an agent can complete change across model releases when difficulty is calibrated by successful human completion time?

Design and assumptions

The paper combines HCAST software tasks, RE-Bench research engineering tasks and short software atomic actions. Human baselines establish task durations, with researcher estimates for some tasks. Continuous scores are converted to task-specific binary success thresholds. A logistic model relates success to log human time; its midpoint defines the 50% horizon. Family weighting prevents large task families dominating the fit.

Source findings

The paper finds a strong relation between human task duration and agent success, and increasing estimated horizons across the examined releases (§§2.3–3).

Robustness checks include retrodiction, SWE-bench Verified, task messiness and internal pull requests. Messier tasks reduce absolute performance; human context changes the duration baseline (§4).

Table 1 includes a JSON transformation task inferred from input/output examples. Its automatic check illustrates the suite’s measurable endpoints while also showing the constraint imposed by machine scoring.

NUWA interpretation

A longer horizon supports greater capability on this software distribution. It does not by itself demonstrate sustained autonomy, replication, willingness to resist intervention or a particular economic impact. The source’s extrapolations remain conditional on trend continuation and generalization.

Limits and contrary evidence

Task durations depend on skill and context; internal maintainers and contractors can take very different times. Automatically scored, mostly static tasks omit many resource constraints, interactions and costly mistakes. Continuous-score thresholds and family weighting are modeling choices. The original paper and live 1.1 dashboard must not be silently treated as one unchanged suite.

Research connections

RE-Bench contributes tasks and human baselines to the measurement design (§2.1). RepliBench and shutdown resistance examine distinct requirements and behaviors; connecting them to this horizon requires a new same-system evidence argument, not score averaging.

Research and decision relevance

Model recommendation: use horizons to select task difficulty and desired reliability, attach the exact suite version and interval, and request separate evidence for intervention response.