{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"hcast","title":"HCAST: Human-Calibrated Autonomy Software Tasks","aliases":["HCAST"],"authors":["David Rein","Joel Becker","Amy Deng","Seraphina Nix","Chris Canal","Daniel O'Connel","Pip Arnott","Ryan Bloom","Thomas Broadley","Katharyn Garcia","Brian Goodrich","Max Hasin","Sami Jawhar","Megan Kinniment","Thomas Kwa","Aron Lajko","Nate Rush","Lucas Jun Koba Sato","Sydney Von Arx","Ben West","Lawrence Chan","Elizabeth Barnes"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"hcast-publication","attributionSourceId":"hcast-publication-author-source","originalUrl":"https://arxiv.org/abs/2503.17354","publicationDate":"2025-03-21","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.","zh":"主维度依据原协议的任务结果确定。主题关联见研究档案，不推断尚未审阅的问题级关联。"},"protocolIds":["hcast"],"contributions":[{"organizationId":"metr","role":"research","sourceId":"hcast-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"equistamp","role":"research","sourceId":"hcast-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Uses human-calibrated software tasks to relate agent task completion to measured human effort.","zh":"使用经过人类校准的软件任务，将智能体完成情况与人类投入关联。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"hcast","title":"HCAST: Human-Calibrated Autonomy Software Tasks","en":{"question":"Can human task duration ground autonomy capability measurement?","design":"189 software, ML, cyber and reasoning tasks include 563 human attempts totaling over 1,500 hours, with manual task quality assurance.","findings":["Agents succeeded 70–80% on sub-hour tasks and below 20% on tasks estimated above four hours.","Some task-time estimates rely on forecasts rather than successful human attempts (§3.2)."],"interpretation":"NUWA interpretation: task-time calibration helps contextualize success, but observed and forecast durations are different evidence.","limits":"Long-task human baselines face selection and logistical biases; the tasks incompletely represent real work (§5).","decisionRelevance":"Model recommendation: retain time-estimate source and task family when interpreting autonomy horizons.","takeaway":"NUWA interpretation: task-time calibration helps contextualize success, but observed and forecast durations are different evidence.","connections":[],"scoring":"Task success related to observed or forecast human time-to-complete.","conditions":"189 tasks and 563 human attempts; forecast-based times more common for longer tasks.","statistics":"Agents succeeded 70–80% on sub-hour tasks and below 20% on tasks estimated above four hours. Some task-time estimates rely on forecasts rather than successful human attempts (§3.2).","usage":"Model recommendation: retain time-estimate source and task family when interpreting autonomy horizons."},"zh":{"question":"人类任务时长能否为自主能力测量提供锚点？","design":"189项软件、机器学习、网络与推理任务收集563次人类尝试，累计逾1,500小时，并人工质检。","findings":["智能体在一小时内任务成功70–80%，四小时以上任务低于20%。","部分时长源于预测，并非成功人类尝试（§3.2）。"],"interpretation":"NUWA 解读：时长校准有助解释成功，但观测和预测是不同证据。","limits":"长任务基线有选择与招募偏差，且任务不能完全代表真实工作（§5）。","decisionRelevance":"模型建议：解释自主时长时保留估时来源与任务族。","takeaway":"NUWA 解读：时长校准有助解释成功，但观测和预测是不同证据。","connections":[],"scoring":"任务成功对应观测或预测的人类完成时长。","conditions":"189任务、563人类尝试，长任务更常依赖预测。","statistics":"智能体在一小时内任务成功70–80%，四小时以上任务低于20%。 部分时长源于预测，并非成功人类尝试（§3.2）。","usage":"模型建议：解释自主时长时保留估时来源与任务族。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"hcast-publication-author-source","url":"https://arxiv.org/html/2503.17354","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established","artifactSha256":"74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8"},{"sourceId":"hcast-publication-author-source","url":"https://arxiv.org/html/2503.17354","locator":"§2 Benchmark Creation Process","version":"Selected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established","artifactSha256":"74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8"},{"sourceId":"hcast-publication-author-source","url":"https://arxiv.org/html/2503.17354","locator":"§3.2 Human Time Estimates","version":"Selected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established","artifactSha256":"74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8"},{"sourceId":"hcast-publication-author-source","url":"https://arxiv.org/html/2503.17354","locator":"§5 Discussion","version":"Selected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established","artifactSha256":"74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"58ecb8034bd0e0b9541b5374a304aff627f525d039276c1fed60b26f679b7ff3"}},"sources":[{"id":"hcast-publication","title":"HCAST: Human-Calibrated Autonomy Software Tasks","url":"https://arxiv.org/abs/2503.17354","version":"arXiv:2503.17354v1","publicationDate":"2025-03-21","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2503.17354","authors":["David Rein","Joel Becker","Amy Deng","Seraphina Nix","Chris Canal","Daniel O’Connell","Pip Arnott","Ryan Bloom","Thomas Broadley","Katharyn Garcia","Brian Goodrich","Max Hasin","Sami Jawhar","Megan Kinniment","Thomas Kwa","Aron Lajko","Nate Rush","Lucas Jun Koba Sato","Sydney Von Arx","Ben West","Lawrence Chan","Elizabeth Barnes"],"aliases":["https://doi.org/10.48550/arXiv.2503.17354","https://arxiv.org/abs/2503.17354","https://arxiv.org/pdf/2503.17354"]}]}