NUWA 风险评测 · 评测基准
PaperBench: Evaluating AI's Ability to Replicate AI Research
NUWA 研究解读 · 独立复核待完成
NUWA Research Note
研究问题研究设计与假设来源发现NUWA 解读局限与相反证据研究与决策意义机构与角色原始资源模型辅助整理 · 选读原始全文或官方文章中的相关章节；尚未完成逐章节审计。
研究速读
NUWA 解读：按量表计的部分复现得分不同于完整复现论文比例。
研究问题
智能体能否从零复现机器学习论文，而非只写貌似合理的实现？
研究设计与假设
20篇ICML 2024论文分解为8,316项要求，量表与原作者共同制定，并用JudgeEval核对模型判分。
来源发现
主要实验中Claude 3.5 Sonnet(New)平均复现得分21.0%，o1为13.2%。
博士人类基线仅覆盖部分论文。
NUWA 解读
NUWA 解读：按量表计的部分复现得分不同于完整复现论文比例。
局限与相反证据
论文选择、自动判分和部分人类基线限制广泛研究自动化结论。
研究与决策意义
模型建议：复用时区分代码开发要求、实际实验执行及完整复现。
精确来源位置
- PaperBench: Evaluating AI's Ability to Replicate AI Research · AbstractSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
- PaperBench: Evaluating AI's Ability to Replicate AI Research · §3 DatasetSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
- PaperBench: Evaluating AI's Ability to Replicate AI Research · §4 LLM JudgeSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
- PaperBench: Evaluating AI's Ability to Replicate AI Research · §5.2 Main ExperimentSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
- PaperBench: Evaluating AI's Ability to Replicate AI Research · §5.4 Human Baseline PerformanceSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
来源核查、数据提取、编辑批准与实验复现分别记录。 方法说明 →
机构与角色
作者 (13)
Giulio Starace · Oliver Jaffe · Dane Sherburn · James Aung · Jun Shern Chan · Leon Maksin · Rachel Dias · Evan Mays · Benjamin Kinsella · Wyatt Thompson · Johannes Heidecke · Amelia Glaese · Tejal Patwardhan
评测协议
引用与复用
引用研究发现时请引用原始出版物；引用 NUWA 的整理与解释时，单独引用本站记录及对应快照。
原始出版物 · BibTeX
@misc{paperbench,
title = {PaperBench: Evaluating AI's Ability to Replicate AI Research},
url = {https://arxiv.org/abs/2504.01848v3},
author = {Giulio Starace and Oliver Jaffe and Dane Sherburn and James Aung and Jun Shern Chan and Leon Maksin and Rachel Dias and Evan Mays and Benjamin Kinsella and Wyatt Thompson and Johannes Heidecke and Amelia Glaese and Tejal Patwardhan},
year = {2025},
date = {2025-04-02},
version = {arXiv:2504.01848v3},
note = {arXiv:2504.01848v3}
}
NUWA 记录 · BibTeX
@misc{paperbench-nuwa,
title = {PaperBench: Evaluating AI's Ability to Replicate AI Research — NUWA research record},
url = {https://whitzard.tech/nuwa/whitzard-index/work/paperbench/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--paperbench.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--paperbench.json.}
}
证据快照 2026-10-29.1 · 索引更新 · RSS
参考文献
- [1] Starace, Giulio, Jaffe, Oliver 等. PaperBench: Evaluating AI's Ability to Replicate AI Research. 2025-04-07. v3 ; Author affiliations on the original paper title page; Abstract; §3 Dataset