{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"instrumentaleval","title":"Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?","aliases":[],"authors":["Yufei He","Yuexin Li","Jiaying Wu","Yuan Sui","Yulin Chen","Bryan Hooi"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"instrumentaleval-publication","attributionSourceId":"instrumentaleval-publication-author-source","originalUrl":"https://arxiv.org/abs/2502.12206","publicationDate":"2025-02-16","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":["instrumentaleval"],"contributions":[{"organizationId":"nus","role":"research","sourceId":"instrumentaleval-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Tests model responses for instrumental goals such as resource acquisition and self-preservation under specified scenarios.","zh":"在指定场景下测试模型回答中的资源获取、自我维持等工具性目标。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"instrumentaleval","title":"Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?","en":{"question":"When do goal-directed model responses exhibit instrumental-goal behaviors?","design":"The benchmark varies prior task, model and prompt design and uses automated judges to label instrumental convergence.","findings":["The authors report higher labeled convergence for the studied direct-RL cohort and greater rates with goal nudging.","Judge selection significantly changes detection (§4.6)."],"interpretation":"NUWA interpretation: observed cohort associations do not isolate RL as a causal training effect.","limits":"Training histories and capabilities differ across model groups; judge sensitivity and constructed tasks limit deployment inference.","decisionRelevance":"Model recommendation: retain judge identity and nudging condition, and avoid causal claims from unmatched model cohorts.","takeaway":"NUWA interpretation: observed cohort associations do not isolate RL as a causal training effect.","connections":[],"scoring":"Automated judge labels for instrumental-goal responses, sensitive to judge selection.","conditions":"Prior task, model cohort and explicit goal nudging vary.","statistics":"Judge selection significantly changes detection (§4.6).","usage":"Model recommendation: retain judge identity and nudging condition, and avoid causal claims from unmatched model cohorts."},"zh":{"question":"目标驱动回答何时出现工具性目标行为？","design":"基准改变前置任务、模型与提示，以自动判官标注工具趋同。","findings":["作者报告受测直接RL组较高，目标强调也增加标注行为。","判官选择显著影响检出（§4.6）。"],"interpretation":"NUWA 解读：群组关联不能隔离RL训练的因果效应。","limits":"各组训练史与能力不同，判官敏感性和构造任务限制部署推断。","decisionRelevance":"模型建议：保留判官与目标强调条件，避免据未匹配组作因果结论。","takeaway":"NUWA 解读：群组关联不能隔离RL训练的因果效应。","connections":[],"scoring":"自动判工具性目标回答，受判官选择影响。","conditions":"改变前置任务、模型组与目标强调。","statistics":"判官选择显著影响检出（§4.6）。","usage":"模型建议：保留判官与目标强调条件，避免据未匹配组作因果结论。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"instrumentaleval-publication-author-source","url":"https://arxiv.org/html/2502.12206","locator":"§3 InstrumentalEval","version":"Selected original resource https://arxiv.org/html/2502.12206; sha256:1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1; archived source capture; exact capture time not independently established","artifactSha256":"1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1"},{"sourceId":"instrumentaleval-publication-author-source","url":"https://arxiv.org/html/2502.12206","locator":"§4.5 Explicit Goal Nudging Increases Instrumental Convergence","version":"Selected original resource https://arxiv.org/html/2502.12206; sha256:1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1; archived source capture; exact capture time not independently established","artifactSha256":"1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1"},{"sourceId":"instrumentaleval-publication-author-source","url":"https://arxiv.org/html/2502.12206","locator":"§4.6 Judge Model Selection Significantly Impacts Convergence Detection","version":"Selected original resource https://arxiv.org/html/2502.12206; sha256:1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1; archived source capture; exact capture time not independently established","artifactSha256":"1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1"},{"sourceId":"instrumentaleval-publication-author-source","url":"https://arxiv.org/html/2502.12206","locator":"Limitations","version":"Selected original resource https://arxiv.org/html/2502.12206; sha256:1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1; archived source capture; exact capture time not independently established","artifactSha256":"1bb93b48504f755173f8730476788c3add572c14d95d32d254b8f7594d7803f1"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"60b97da62344423cf9a613822b854733fb8d3aa4c6504b327733f6cf2258c1c1"}},"sources":[{"id":"instrumentaleval-publication","title":"Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?","url":"https://arxiv.org/abs/2502.12206","version":"arXiv:2502.12206v1","publicationDate":"2025-02-16","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2502.12206","authors":["Yufei He","Yuexin Li","Jiaying Wu","Yuan Sui","Yulin Chen","Bryan Hooi"],"aliases":["https://doi.org/10.48550/arXiv.2502.12206","https://arxiv.org/abs/2502.12206","https://arxiv.org/pdf/2502.12206"]}]}