风险评测 · 评测基准

HCAST: Human-Calibrated Autonomy Software Tasks

NUWA 研究解读 · 独立复核待完成

首次发表: 2025-03-21来源核验: 2026-10-25

原始来源 ↗引用本记录 ↓自主性 →

NUWA Research Note

研究问题研究设计与假设来源发现 NUWA 解读局限与相反证据研究与决策意义机构与角色原始资源

模型辅助整理 · 选读原始全文或官方文章中的相关章节；尚未完成逐章节审计。

研究速读

NUWA 解读：时长校准有助解释成功，但观测和预测是不同证据。

研究问题

人类任务时长能否为自主能力测量提供锚点？

研究设计与假设

189项软件、机器学习、网络与推理任务收集563次人类尝试，累计逾1,500小时，并人工质检。

HCAST — Figure 6 — Figure 6 · 原文 Figure 6。请结合下方的协议范围与局限阅读。原图与来源 ↗

来源发现

智能体在一小时内任务成功70–80%，四小时以上任务低于20%。

部分时长源于预测，并非成功人类尝试（§3.2）。

NUWA 解读

NUWA 解读：时长校准有助解释成功，但观测和预测是不同证据。

局限与相反证据

长任务基线有选择与招募偏差，且任务不能完全代表真实工作（§5）。

研究与决策意义

模型建议：解释自主时长时保留估时来源与任务族。

精确来源位置

HCAST: Human-Calibrated Autonomy Software Tasks · AbstractSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
HCAST: Human-Calibrated Autonomy Software Tasks · §2 Benchmark Creation ProcessSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
HCAST: Human-Calibrated Autonomy Software Tasks · §3.2 Human Time EstimatesSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established
HCAST: Human-Calibrated Autonomy Software Tasks · §5 DiscussionSelected original resource https://arxiv.org/html/2503.17354; sha256:74636576fcb11fa78c419dd47c2da316ac7d432a3ab57bff64b51cdf94a683a8; archived source capture; exact capture time not independently established

来源核查、数据提取、编辑批准与实验复现分别记录。方法说明 →

机构与角色

研究机构 METR Equistamp

作者 (22)

David Rein · Joel Becker · Amy Deng · Seraphina Nix · Chris Canal · Daniel O'Connel · Pip Arnott · Ryan Bloom · Thomas Broadley · Katharyn Garcia · Brian Goodrich · Max Hasin · Sami Jawhar · Megan Kinniment · Thomas Kwa · Aron Lajko · Nate Rush · Lucas Jun Koba Sato · Sydney Von Arx · Ben West · Lawrence Chan · Elizabeth Barnes

评测协议

HCAST →

引用与复用

引用研究发现时请引用原始出版物；引用 NUWA 的整理与解释时，单独引用本站记录及对应快照。

BibTeX ↓RIS ↓记录与来源元数据 · JSON ↓阅读覆盖 · JSON 提交更正

原始出版物 · BibTeX

@misc{hcast,
  title = {HCAST: Human-Calibrated Autonomy Software Tasks},
  url = {https://arxiv.org/abs/2503.17354},
  author = {David Rein and Joel Becker and Amy Deng and Seraphina Nix and Chris Canal and Daniel O'Connel and Pip Arnott and Ryan Bloom and Thomas Broadley and Katharyn Garcia and Brian Goodrich and Max Hasin and Sami Jawhar and Megan Kinniment and Thomas Kwa and Aron Lajko and Nate Rush and Lucas Jun Koba Sato and Sydney Von Arx and Ben West and Lawrence Chan and Elizabeth Barnes},
  year = {2025},
  date = {2025-03-21},
  version = {arXiv:2503.17354v1},
  note = {arXiv:2503.17354v1}
}

NUWA 记录 · BibTeX

@misc{hcast-nuwa,
  title = {HCAST: Human-Calibrated Autonomy Software Tasks — NUWA research record},
  url = {https://whitzard.tech/nuwa/whitzard-index/work/hcast/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--hcast.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--hcast.json.}
}

证据快照 2026-10-29.1 · 索引更新 · RSS

参考文献

[1] David Rein, Joel Becker 等. HCAST: Human-Calibrated Autonomy Software Tasks. 2025-03-21. v1 [arXiv]; Author affiliations on the original paper title page; Abstract; §2 Benchmark Creation Process