METR Time Horizons

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

How does the difficulty of software tasks an agent can complete change across model releases when difficulty is calibrated by successful human completion time?

Task population

The paper combines HCAST software tasks, RE-Bench research engineering tasks and short software atomic actions. Human baselines establish task durations, with researcher estimates for some tasks. Continuous scores are converted to task-specific binary success thresholds. A logistic model relates success to log human time; its midpoint defines the 50% horizon. Family weighting prevents large task families dominating the fit.

Scoring and failure modes

The fitted human-duration threshold at a specified success probability is the endpoint. The 50% and 80% horizons answer different reliability questions. Scores and uncertainty for later METR 1.1 releases belong to their separately versioned source series.

System conditions

The paper combines HCAST software tasks, RE-Bench research engineering tasks and short software atomic actions. Human baselines establish task durations, with researcher estimates for some tasks. Continuous scores are converted to task-specific binary success thresholds. A logistic model relates success to log human time; its midpoint defines the 50% horizon. Family weighting prevents large task families dominating the fit.

Statistical meaning

Logistic fitting uses human-duration estimates and family weighting; the original paper’s intervals and the dashboard’s later uncertainty must retain their own versioned methods.

Validity and interpretation

Task durations depend on skill and context; internal maintainers and contractors can take very different times. Automatically scored, mostly static tasks omit many resource constraints, interactions and costly mistakes. Continuous-score thresholds and family weighting are modeling choices. The original paper and live 1.1 dashboard must not be silently treated as one unchanged suite.

Appropriate research use

Model recommendation: use horizons to select task difficulty and desired reliability, attach the exact suite version and interval, and request separate evidence for intervention response.

• Measuring AI Ability to Complete Long Software Tasks · §2.1 Task suite / dataset; Table 1arXiv:2503.14499v4 (2026-10-25 capture)
• Measuring AI Ability to Complete Long Software Tasks · §2.2 BaseliningarXiv:2503.14499v4 (2026-10-25 capture)
• Measuring AI Ability to Complete Long Software Tasks · §3.1 Computing time horizon; Figure 4arXiv:2503.14499v4 (2026-10-25 capture)
• Measuring AI Ability to Complete Long Software Tasks · §4 External validity and robustnessarXiv:2503.14499v4 (2026-10-25 capture)
• Measuring AI Ability to Complete Long Software Tasks · §6 Discussion; Appendix B.2; Appendix E.3arXiv:2503.14499v4 (2026-10-25 capture)