{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"time-horizons","title":"Task-Completion Time Horizons of Frontier AI Models","aliases":["METR Time Horizons"],"authors":["Thomas Kwa","Ben West","Joel Becker","Amy Deng","Katharyn Garcia","Max Hasin","Sami Jawhar","Megan Kinniment","Nate Rush","Sydney Von Arx","Ryan Bloom","Thomas Broadley","Haoxing Du","Brian Goodrich","Nikola Jurkovic","Luke Harold Miles","Seraphina Nix","Tao Lin","Chris Painter","Neev Parikh","David Rein","Lucas Jun Koba Sato","Hjalmar Wijk","Daniel M. Ziegler","Elizabeth Barnes","Lawrence Chan"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"time-horizons","attributionSourceId":"time-horizons-author-source","originalUrl":"https://metr.org/time-horizons/","publicationDate":"2026-09-08","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.","zh":"主维度依据来源实际测量的构念；具体指标可能属于其他维度。"},"protocolIds":["time-horizons"],"contributions":[{"organizationId":"metr","role":"research","sourceId":"time-horizons-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"anthropic","role":"research","sourceId":"time-horizons-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Task difficulty expressed in human expert completion time.","zh":"以人类专家完成时间表示的任务难度。"},"note":{"question":{"en":"Fitted success probability across a software-task distribution.","zh":"软件任务分布上的拟合成功概率。"},"method":{"en":"AI agent completing software tasks under a harness","zh":"在脚手架下完成软件任务的 AI 智能体"},"finding":{"en":"Task difficulty expressed in human expert completion time.","zh":"以人类专家完成时间表示的任务难度。"},"supports":null,"limits":{"en":"The public page is no longer actively updated; time horizon is not agent runtime or a universal automation measure.","zh":"该公开页面已停止持续更新；时间范围不是智能体运行时长或通用自动化指标。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"time-horizons","title":"Task-Completion Time Horizons of Frontier AI Models","en":{"takeaway":"Time horizon connects task success to the time a human expert needs for a task. It is a fitted property of a task distribution, not a stopwatch on an autonomous agent.","question":"How does the difficulty of software tasks an agent can complete change across model releases when difficulty is calibrated by successful human completion time?","design":"The paper combines HCAST software tasks, RE-Bench research engineering tasks and short software atomic actions. Human baselines establish task durations, with researcher estimates for some tasks. Continuous scores are converted to task-specific binary success thresholds. A logistic model relates success to log human time; its midpoint defines the 50% horizon. Family weighting prevents large task families dominating the fit.","scoring":"The fitted human-duration threshold at a specified success probability is the endpoint. The 50% and 80% horizons answer different reliability questions. Scores and uncertainty for later METR 1.1 releases belong to their separately versioned source series.","findings":["The paper finds a strong relation between human task duration and agent success, and increasing estimated horizons across the examined releases (§§2.3–3).","Robustness checks include retrodiction, SWE-bench Verified, task messiness and internal pull requests. Messier tasks reduce absolute performance; human context changes the duration baseline (§4).","Table 1 includes a JSON transformation task inferred from input/output examples. Its automatic check illustrates the suite’s measurable endpoints while also showing the constraint imposed by machine scoring."],"interpretation":"A longer horizon supports greater capability on this software distribution. It does not by itself demonstrate sustained autonomy, replication, willingness to resist intervention or a particular economic impact. The source’s extrapolations remain conditional on trend continuation and generalization.","limits":"Task durations depend on skill and context; internal maintainers and contractors can take very different times. Automatically scored, mostly static tasks omit many resource constraints, interactions and costly mistakes. Continuous-score thresholds and family weighting are modeling choices. The original paper and live 1.1 dashboard must not be silently treated as one unchanged suite.","connections":["RE-Bench contributes tasks and human baselines to the measurement design (§2.1). RepliBench and shutdown resistance examine distinct requirements and behaviors; connecting them to this horizon requires a new same-system evidence argument, not score averaging."],"decisionRelevance":"Model recommendation: use horizons to select task difficulty and desired reliability, attach the exact suite version and interval, and request separate evidence for intervention response.","statistics":"Logistic fitting uses human-duration estimates and family weighting; the original paper’s intervals and the dashboard’s later uncertainty must retain their own versioned methods."},"zh":{"takeaway":"任务跨度把成功率与人类专家完成任务所需时间联系起来。它是特定任务分布上的拟合量，并非智能体连续自主运行的计时。","question":"以成功的人类基线用时衡量难度时，不同模型版本能够完成的软件任务范围如何变化？","design":"论文结合 HCAST 软件任务、RE-Bench 研究工程任务和软件原子操作。多数任务由人类基线确定用时，部分任务使用研究者估计。连续评分按任务阈值转为二元成功；逻辑回归把成功率与人类用时的对数关联，其中点定义 50% 任务跨度，并按任务族加权。","scoring":"终点是在给定成功概率下拟合的人类任务时长。50% 与 80% 跨度对应不同可靠性要求；后续 METR 1.1 模型的结果和不确定性属于独立版本的来源序列。","findings":["论文观察到人类任务时长与智能体成功率的关系，以及所研究版本间任务跨度的上升（§§2.3–3）。","稳健性检查包含回溯预测、SWE-bench Verified、任务混杂程度和内部 PR；较混杂任务的绝对表现较低，人类背景知识也影响用时基线（§4）。","表 1 的 JSON 转换任务要求从输入输出示例推断规则；自动验收既使结果可测量，也体现了机器评分对任务开放性的约束。"],"interpretation":"较长跨度支持该软件任务分布中的能力提升，但不直接证明持续自主性、复制能力、抗拒干预或经济影响。论文外推依赖趋势延续与分布泛化的条件。","limits":"用时取决于专业经验和上下文；维护者与外部承包者的时长可能明显不同。自动评分、主要静态的环境缺少许多资源约束、交互和高代价错误。阈值、加权均是建模选择，原论文与后续 1.1 看板不能视为完全相同的任务集。","connections":["RE-Bench 为该设计提供任务和人类基线（§2.1）。RepliBench 与关机抗拒测量其他条件和行为；与跨度建立联系需要同一系统的证据论证，不能平均分数。"],"decisionRelevance":"模型建议：用跨度选择任务难度及可靠性目标，附上任务集版本与区间，并独立核查干预响应。","statistics":"逻辑回归依赖人类时长与任务族加权；原论文区间和后续看板的不确定性分别保留各自版本方法。"},"figureRefs":[{"figureId":"time-horizons-figure","sourceId":"time-horizons-figure-source","locator":"Figure 3","original":"/images/nuwa-index/time-horizons-original.png","provenanceUrl":"https://arxiv.org/html/2503.14499v4/plots/bar_chart_weighted_scores/headline.png","disposition":"existing-repository-provenance-not-newly-authorized","review":"Retain existing repository provenance only. Image suitability and rights are not newly approved by this editorial draft."}],"assertionPolicy":{"design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","connections":"nuwa-interpretation-with-source-refs","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"time-horizons-author-source","url":"https://arxiv.org/html/2503.14499","locator":"§2.1 Task suite / dataset; Table 1","version":"arXiv:2503.14499v4 (2026-10-25 capture)"},{"sourceId":"time-horizons-author-source","url":"https://arxiv.org/html/2503.14499","locator":"§2.2 Baselining","version":"arXiv:2503.14499v4 (2026-10-25 capture)"},{"sourceId":"time-horizons-author-source","url":"https://arxiv.org/html/2503.14499","locator":"§3.1 Computing time horizon; Figure 4","version":"arXiv:2503.14499v4 (2026-10-25 capture)"},{"sourceId":"time-horizons-author-source","url":"https://arxiv.org/html/2503.14499","locator":"§4 External validity and robustness","version":"arXiv:2503.14499v4 (2026-10-25 capture)"},{"sourceId":"time-horizons-author-source","url":"https://arxiv.org/html/2503.14499","locator":"§6 Discussion; Appendix B.2; Appendix E.3","version":"arXiv:2503.14499v4 (2026-10-25 capture)"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":"full-text","readingCoverage":"selected-methods-results-limitations-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"7c7a404cbfa9c51d916d1f6894c9e3177d1cbe38c973a2b7ddd419927d6aa73e"}},"sources":[{"id":"time-horizons","title":"Task-Completion Time Horizons of Frontier AI Models","url":"https://metr.org/time-horizons/","version":"Time Horizon 1.1; page updated 2026-09-08","publicationDate":"2026-09-08","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"https://metr.org/time-horizons","authors":["Thomas Kwa","Ben West 1 1 footnotemark: 1","Joel Becker","Amy Deng","Katharyn GarciaMax Hasin , Sami Jawhar , Megan Kinniment , Nate Rush , Sydney Von ArxRyan Bloom , Thomas Broadley , Haoxing Du , Brian Goodrich , Nikola Jurkovic , Luke Harold Miles , Seraphina Nix , Tao Lin , Chris Painter , Neev Parikh , David Rein , Lucas Jun Koba Sato , Hjalmar Wijk , Daniel M. ZieglerElizabeth Barnes , Lawrence Chan Model Evaluation & Threat Research (METR)"]}]}