{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"re-bench","title":"RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts","aliases":["RE-Bench"],"authors":["Hjalmar Wijk","Tao Lin","Joel Becker","Sami Jawhar","Neev Parikh","Thomas Broadley","Lawrence Chan","Michael Chen","Josh Clymer","Jai Dhyani","Elena Ericheva","Katharyn Garcia","Brian Goodrich","Nikola Jurkovic","Holden Karnofsky","Megan Kinniment","Aron Lajko","Seraphina Nix","Lucas Sato","William Saunders","Maksym Taran","Ben West","Elizabeth Barnes"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"re-bench","attributionSourceId":"re-bench-author-source","originalUrl":"https://arxiv.org/abs/2411.15114v2","publicationDate":"2024-11-22","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.","zh":"主维度依据来源实际测量的构念；具体指标可能属于其他维度。"},"protocolIds":["re-bench"],"contributions":[{"organizationId":"metr","role":"research","sourceId":"re-bench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"redwood","role":"research","sourceId":"re-bench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"harvard","role":"research","sourceId":"re-bench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"anthropic","role":"research","sourceId":"re-bench-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Research engineering performance under time budgets.","zh":"时间预算约束下的研究工程表现。"},"note":{"question":{"en":"ML research engineering outcomes relative to human experts.","zh":"相对于人类专家的机器学习研究工程成果。"},"method":{"en":"AI agent performing ML research engineering tasks","zh":"执行机器学习研究工程任务的 AI 智能体"},"finding":{"en":"Research engineering performance under time budgets.","zh":"时间预算约束下的研究工程表现。"},"supports":null,"limits":{"en":"AI R&D performance is an enabling capability, not evidence of harmful self-improvement.","zh":"AI 研发表现属于促成能力，不构成有害自我改进证据。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"re-bench","title":"RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts","en":{"question":"How do agents compare with experts on open-ended ML research engineering?","design":"Seven environments include 71 eight-hour attempts by 61 distinct experts; agent comparisons vary total time, scaffold and best-of-k selection.","findings":["82% of expert attempts earned nonzero scores; 24% matched or exceeded reference solutions.","Agents led at two hours, but humans achieved about twice the top-agent score at 32 aggregate hours in the reported setup."],"interpretation":"NUWA interpretation: the human–agent ordering depends on time allocation and selection, not a single general research ranking.","limits":"Seven environments do not span frontier R&D; limited scaffold iteration could understate agent capability (§6.3).","decisionRelevance":"Model recommendation: cite normalized score, scaffold, total time and attempt aggregation together.","takeaway":"NUWA interpretation: the human–agent ordering depends on time allocation and selection, not a single general research ranking.","connections":[],"scoring":"Environment-specific normalized research-engineering score; total-time/best-of-k comparisons.","conditions":"Seven environments, multiple scaffolds and time budgets; 71 expert eight-hour attempts.","statistics":"82% of expert attempts earned nonzero scores; 24% matched or exceeded reference solutions. Agents led at two hours, but humans achieved about twice the top-agent score at 32 aggregate hours in the reported setup.","usage":"Model recommendation: cite normalized score, scaffold, total time and attempt aggregation together."},"zh":{"question":"开放式机器学习研究工程中，智能体与专家表现如何？","design":"七个环境收集61位专家的71次八小时尝试；比较改变总时间、框架和最佳样本选择。","findings":["82%的专家尝试得分非零，24%达到或超过参考解。","两小时条件下智能体领先，32小时累计条件下人类约为最佳智能体得分两倍。"],"interpretation":"NUWA 解读：排序取决于时间分配和样本选择，不能视为统一研究能力排名。","limits":"七个环境不能覆盖前沿研发；有限框架迭代可能低估智能体能力（§6.3）。","decisionRelevance":"模型建议：同时引用归一化得分、框架、总时间及尝试聚合方式。","takeaway":"NUWA 解读：排序取决于时间分配和样本选择，不能视为统一研究能力排名。","connections":[],"scoring":"环境特定归一化研究工程分，比较总时间及最佳样本。","conditions":"七环境、多框架与时间预算；71次专家八小时尝试。","statistics":"82%的专家尝试得分非零，24%达到或超过参考解。 两小时条件下智能体领先，32小时累计条件下人类约为最佳智能体得分两倍。","usage":"模型建议：同时引用归一化得分、框架、总时间及尝试聚合方式。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"re-bench-author-source","url":"https://arxiv.org/html/2411.15114v2","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established","artifactSha256":"4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58"},{"sourceId":"re-bench-author-source","url":"https://arxiv.org/html/2411.15114v2","locator":"§3.1 Design goals","version":"Selected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established","artifactSha256":"4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58"},{"sourceId":"re-bench-author-source","url":"https://arxiv.org/html/2411.15114v2","locator":"§4 Experiments and results","version":"Selected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established","artifactSha256":"4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58"},{"sourceId":"re-bench-author-source","url":"https://arxiv.org/html/2411.15114v2","locator":"§6.3 Other Limitations","version":"Selected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established","artifactSha256":"4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"751c7933cfa6aae49d4372ec0959f446a696f2efb7a77e4c6c0606e604dbc48c"}},"sources":[{"id":"re-bench","title":"RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts","url":"https://arxiv.org/abs/2411.15114v2","version":"arXiv:2411.15114v2","publicationDate":"2025-05-27","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"arxiv:2411.15114","authors":["Hjalmar Wijk","Tao Lin","Joel Becker","Sami Jawhar","Neev Parikh","Thomas Broadley","Lawrence Chan 2 2 footnotemark: 2","Michael Chen 2 2 footnotemark: 2","Josh Clymer 2 2 footnotemark: 2","Jai Dhyani 2 2 footnotemark: 2","Elena Ericheva 2 2 footnotemark: 2","Katharyn Garcia 2 2 footnotemark: 2","Brian Goodrich 2 2 footnotemark: 2","Nikola Jurkovic 2 2 footnotemark: 2","Holden Karnofsky 2 2 footnotemark: 2","Megan Kinniment 2 2 footnotemark: 2","Aron Lajko 2 2 footnotemark: 2 3 3 footnotemark: 3","Seraphina Nix 2 2 footnotemark: 2","Lucas Sato 2 2 footnotemark: 2","William Saunders 2 2 footnotemark: 2","Maksym Taran 2 2 footnotemark: 2","Ben West 2 2 footnotemark: 2","Elizabeth Barnes","Model Evaluation and Threat Research (METR)"],"aliases":["https://doi.org/10.48550/arXiv.2411.15114","https://arxiv.org/abs/2411.15114","https://arxiv.org/pdf/2411.15114"]}]}