{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"replibench","title":"RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents","aliases":["RepliBench"],"authors":["Sid Black","Asa Cooper Stickland","Jake Pencharz","Oliver Sourbut","Michael Schmatz","Jay Bailey","Ollie Matthews","Ben Millwood","Alex Remedios","Alan Cooney"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"replibench","attributionSourceId":"replibench-author-source","originalUrl":"https://arxiv.org/abs/2504.18565v2","publicationDate":"2025-04-21","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.","zh":"主维度依据来源实际测量的构念；具体指标可能属于其他维度。"},"protocolIds":["replibench"],"contributions":[{"organizationId":"aisi","role":"research","sourceId":"replibench-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Capabilities needed for autonomous replication.","zh":"自主复制所需的能力。"},"note":{"question":{"en":"Task-family performance on components of replication.","zh":"复制组成任务上的表现。"},"method":{"en":"AI agent completing replication components","zh":"完成复制组成任务的 AI 智能体"},"finding":{"en":"Capabilities needed for autonomous replication.","zh":"自主复制所需的能力。"},"supports":null,"limits":{"en":"Component success does not establish autonomous end-to-end proliferation.","zh":"组成任务成功不构成端到端自主扩散证据。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"replibench","title":"RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents","en":{"takeaway":"RepliBench maps component capabilities needed for autonomous replication. Success on the components does not establish an unrestricted, persistent population of self-replicating agents.","question":"Which prerequisites for obtaining resources, deploying successors and retaining access can language-model agents complete under controlled conditions?","design":"Capability tree analysis decomposes replication into logical prerequisites and task families. The initial assumption is an assistant inside a developer environment with command execution and internet access. Individual task variants isolate components. In the recursive replication family, a parent deploys a successor, that successor deploys another, and the later agent is checked on a basic task. Variants distinguish APIs, local weights and inference servers, with or without a reference scaffold.","scoring":"Task-family pass@10 and aggregate scores summarize repeated task attempts; the mean-domain aggregation and capability-tree aggregation are alternatives, not equivalent end-to-end probabilities. In local-weight variants, intercepted requests are forwarded to the model under evaluation to avoid testing the local model’s intelligence.","findings":["Results show uneven capability: resource provisioning can succeed while robust successor deployment, identity verification and realistic security remain bottlenecks (§5.1).","The reference-scaffold and from-scratch variants test different burdens. The recursive family requires more than copying files: the successor must itself continue the task (§3.3).","Appendix D explains how even modest judge false positives can be amplified by pass@10; simulated-person outcomes need particularly careful interpretation."],"interpretation":"The decomposition is useful for identifying protective control points and missing capabilities. Because coordination is removed from most tasks, component performance can overstate integrated ability. No score establishes that a model will choose to replicate.","limits":"The suite lacks fully open-ended end-to-end replication. Money-making interactions often use simulated people, and security conditions are simplified. Coverage is prioritized rather than exhaustive; alternative paths can be missed. The judge may favor related models or be weaker than the agent. Simulated success is not evidence about real victims or real-world persistence.","connections":["The paper explicitly derives its design from capability-tree analysis. Cybersecurity benchmarks cover adjacent capabilities but cannot fill every omitted branch automatically. Shutdown resistance concerns behavior under intervention and must remain a separate endpoint."],"decisionRelevance":"Model recommendation: trace each claimed replication pathway through its required components and unresolved coordination steps; keep API-only success distinct from locally sustained operation.","statistics":"Figure 4 uses hierarchical bootstrap confidence intervals across model scores and variants. Appendix D cautions that pass@k can amplify judge error."},"zh":{"takeaway":"RepliBench 分解自主复制所需的组件能力。组件成功不等于已经形成不受限制、持续存在的自我复制智能体群体。","question":"在受控条件下，语言模型智能体能够完成获取资源、部署后继与保持访问权限的哪些前提？","design":"能力树把复制分解为逻辑前提与任务族；起始假设是开发机构内部具有命令执行和联网能力的助手。多数变体隔离组件。递归复制任务要求父智能体部署后继，后继再部署下一代，并检验后代基本任务；变体区分 API、本地权重与推理服务，以及有无参考脚手架。","scoring":"任务族 pass@10 与总分汇总重复尝试；领域均值和能力树汇总是不同方案，均非端到端成功概率。本地权重变体将请求转发给受测模型，以免把本地模型的智能误当受测对象。","findings":["结果呈现不均衡能力：资源配置可成功，稳定部署后继、身份核验和真实安全条件仍是瓶颈（§5.1）。","有参考脚手架与从零编写承担不同负担；递归任务不仅复制文件，还要求后继继续执行任务（§3.3）。","附录 D 说明 pass@10 会放大判别器假阳性，模拟人的结果尤其需要谨慎解释。"],"interpretation":"分解有助于识别防护控制点和缺失能力，但多数任务排除了协调成本，可能高估整合表现；分数也不能证明模型会自主选择复制。","limits":"缺少完全开放的端到端复制，赚钱交互常由模型模拟人类，安全条件简化；任务覆盖优先考虑质量而非穷尽所有路径。判别器可能有模型家族偏差或弱于被评估者，模拟成功不等于现实受害或长期存续。","connections":["论文以能力树直接驱动设计。网络基准测量相邻能力，但不能自动补齐遗漏分支；关机抗拒则衡量干预中的行为，需独立保留。"],"decisionRelevance":"模型建议：逐一检查复制路径需要的组件及未解决的协调步骤，把仅通过 API 的成功与本地持续运行分开。","statistics":"图 4 使用模型得分与变体的分层自助法区间；附录 D 警告 pass@k 放大判别器误差。"},"figureRefs":[],"assertionPolicy":{"design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","connections":"nuwa-interpretation-with-source-refs","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"replibench-author-source","url":"https://arxiv.org/html/2504.18565v2","locator":"§3.1 Capability tree analysis; Figure 2","version":"arXiv:2504.18565v2"},{"sourceId":"replibench-author-source","url":"https://arxiv.org/html/2504.18565v2","locator":"§3.2 Evaluation Overview; Table 1","version":"arXiv:2504.18565v2"},{"sourceId":"replibench-author-source","url":"https://arxiv.org/html/2504.18565v2","locator":"§3.3 The Recursive Replication task family; Figure 3","version":"arXiv:2504.18565v2"},{"sourceId":"replibench-author-source","url":"https://arxiv.org/html/2504.18565v2","locator":"§5.1 Overall results; Table 2","version":"arXiv:2504.18565v2"},{"sourceId":"replibench-author-source","url":"https://arxiv.org/html/2504.18565v2","locator":"§6 Limitations; Appendix D Limitations of judge models","version":"arXiv:2504.18565v2"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":"full-text","readingCoverage":"selected-methods-results-limitations-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"e5af0fa6246d2f5bb61788854a0d1ffec9b599dfb4b3e0b7e17c5484c79ee9ab"}},"sources":[{"id":"replibench","title":"RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents","url":"https://arxiv.org/abs/2504.18565v2","version":"arXiv:2504.18565v2","publicationDate":"2025-05-05","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"arxiv:2504.18565","authors":["Sid Black *","Asa Cooper Stickland *","Jake Pencharz","Oliver Sourbut"],"aliases":["https://doi.org/10.48550/arXiv.2504.18565","https://arxiv.org/abs/2504.18565","https://arxiv.org/pdf/2504.18565"]}]}