{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"paperbench","title":"PaperBench: Evaluating AI's Ability to Replicate AI Research","aliases":["PaperBench"],"authors":["Giulio Starace","Oliver Jaffe","Dane Sherburn","James Aung","Jun Shern Chan","Leon Maksin","Rachel Dias","Evan Mays","Benjamin Kinsella","Wyatt Thompson","Johannes Heidecke","Amelia Glaese","Tejal Patwardhan"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"paperbench","attributionSourceId":"paperbench","originalUrl":"https://arxiv.org/abs/2504.01848v3","publicationDate":"2025-04-02","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Primary dimension follows graded task completion.","zh":"主维度依据经评分的任务完成结果。"},"protocolIds":["paperbench"],"contributions":[{"organizationId":"openai","role":"research","sourceId":"paperbench","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Evaluates end-to-end replication of AI research papers using hierarchical rubrics.","zh":"使用分层评分规则评测 AI 研究论文的端到端复现。"},"note":{"question":{"en":"Rubric-graded research replication tasks with an agent scaffold.","zh":"在智能体脚手架下完成基于评分规则的研究复现任务。"},"method":{"en":"Research agents replicating published ML papers","zh":"复现已发表机器学习论文的研究智能体"},"finding":{"en":"Evaluates end-to-end replication of AI research papers using hierarchical rubrics.","zh":"使用分层评分规则评测 AI 研究论文的端到端复现。"},"supports":{"en":"Primary dimension follows graded task completion.","zh":"主维度依据经评分的任务完成结果。"},"limits":{"en":"AI research replication is an enabling capability, not evidence of unauthorized autonomous action.","zh":"AI 研究复现是前提能力证据，不证明未经授权的自主行动。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"paperbench","title":"PaperBench: Evaluating AI's Ability to Replicate AI Research","en":{"question":"Can agents reproduce machine-learning papers from scratch rather than merely write plausible implementations?","design":"Twenty ICML 2024 papers are decomposed into 8,316 graded requirements, with author-developed rubrics and an LLM judge tested on JudgeEval.","findings":["Claude 3.5 Sonnet (New) obtained a mean replication score of 21.0%; o1 obtained 13.2% in the main experiment.","Human PhD baselines were collected on a subset, not the complete paper population."],"interpretation":"NUWA interpretation: rubric-weighted partial replication is distinct from the fraction of papers fully replicated.","limits":"Paper selection, automated grading and subset-only human baselines constrain broad research-automation claims.","decisionRelevance":"Model recommendation: distinguish code-development requirements, executed experiments and full replication before reuse.","takeaway":"NUWA interpretation: rubric-weighted partial replication is distinct from the fraction of papers fully replicated.","connections":[],"scoring":"Rubric-weighted Replication Score over 8,316 requirements, not binary whole-paper replication.","conditions":"Twenty ICML 2024 papers; automated judge and subset-only PhD baselines.","statistics":"Claude 3.5 Sonnet (New) obtained a mean replication score of 21.0%; o1 obtained 13.2% in the main experiment.","usage":"Model recommendation: distinguish code-development requirements, executed experiments and full replication before reuse."},"zh":{"question":"智能体能否从零复现机器学习论文，而非只写貌似合理的实现？","design":"20篇ICML 2024论文分解为8,316项要求，量表与原作者共同制定，并用JudgeEval核对模型判分。","findings":["主要实验中Claude 3.5 Sonnet(New)平均复现得分21.0%，o1为13.2%。","博士人类基线仅覆盖部分论文。"],"interpretation":"NUWA 解读：按量表计的部分复现得分不同于完整复现论文比例。","limits":"论文选择、自动判分和部分人类基线限制广泛研究自动化结论。","decisionRelevance":"模型建议：复用时区分代码开发要求、实际实验执行及完整复现。","takeaway":"NUWA 解读：按量表计的部分复现得分不同于完整复现论文比例。","connections":[],"scoring":"按8,316要求加权的复现分，不是整篇复现二元比例。","conditions":"20篇ICML 2024，模型判分及仅部分博士基线。","statistics":"主要实验中Claude 3.5 Sonnet(New)平均复现得分21.0%，o1为13.2%。","usage":"模型建议：复用时区分代码开发要求、实际实验执行及完整复现。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"paperbench","url":"https://arxiv.org/html/2504.01848v3","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established","artifactSha256":"405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7"},{"sourceId":"paperbench","url":"https://arxiv.org/html/2504.01848v3","locator":"§3 Dataset","version":"Selected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established","artifactSha256":"405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7"},{"sourceId":"paperbench","url":"https://arxiv.org/html/2504.01848v3","locator":"§4 LLM Judge","version":"Selected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established","artifactSha256":"405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7"},{"sourceId":"paperbench","url":"https://arxiv.org/html/2504.01848v3","locator":"§5.2 Main Experiment","version":"Selected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established","artifactSha256":"405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7"},{"sourceId":"paperbench","url":"https://arxiv.org/html/2504.01848v3","locator":"§5.4 Human Baseline Performance","version":"Selected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established","artifactSha256":"405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"b5a244d54e090f23e2cfafaa9cc7d2c569f5c155cb47188e7ff9eec53e3e44af"}},"sources":[{"id":"paperbench","title":"PaperBench: Evaluating AI's Ability to Replicate AI Research","url":"https://arxiv.org/abs/2504.01848v3","version":"arXiv:2504.01848v3","publicationDate":"2025-04-07","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-25","documentId":"arxiv:2504.01848","authors":["Starace, Giulio","Jaffe, Oliver","Sherburn, Dane","Aung, James","Chan, Jun Shern","Maksin, Leon","Dias, Rachel","Mays, Evan","Kinsella, Benjamin","Thompson, Wyatt","Heidecke, Johannes","Glaese, Amelia","Patwardhan, Tejal"],"aliases":["https://doi.org/10.48550/arXiv.2504.01848","https://arxiv.org/abs/2504.01848","https://arxiv.org/pdf/2504.01848"]}]}