{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"shade-arena","title":"SHADE-Arena: Evaluating sabotage and monitoring in LLM agents","aliases":["SHADE-Arena"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"shade-arena","attributionSourceId":"shade-arena","originalUrl":"https://www.anthropic.com/research/shade-arena-sabotage-monitoring","publicationDate":"2025-06-16","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Task completion is the primary dimension; monitor detection is a safeguard measure and is not a second risk score.","zh":"任务完成是主维度；监测发现率属于防护测量，并非第二项风险分数。"},"protocolIds":["shade-arena"],"contributions":[{"organizationId":"anthropic","role":"publisher","sourceId":"shade-arena","locator":"Official publication masthead and article byline"}],"summary":{"en":"Pairs benign multi-step tasks with explicitly assigned covert side tasks and monitoring.","zh":"将多步骤良性任务与明确指派的隐蔽副任务及监测配对。"},"note":{"question":{"en":"Covert side-task completion and monitor detection under defined tool access.","zh":"在确定工具权限下测量隐蔽副任务完成及监测发现情况。"},"method":{"en":"Agents assigned a covert side task alongside a legitimate task","zh":"同时被指派隐蔽副任务与正当任务的智能体"},"finding":{"en":"Pairs benign multi-step tasks with explicitly assigned covert side tasks and monitoring.","zh":"将多步骤良性任务与明确指派的隐蔽副任务及监测配对。"},"supports":{"en":"Task completion is the primary dimension; monitor detection is a safeguard measure and is not a second risk score.","zh":"任务完成是主维度；监测发现率属于防护测量，并非第二项风险分数。"},"limits":{"en":"An assigned malicious side task measures ability to execute it, not spontaneous desire to sabotage.","zh":"被指派的恶意副任务测量执行能力，不证明自发破坏倾向。"},"connections":null},"resources":[{"label":"Read the paper","url":"https://www.arxiv.org/abs/2506.15740"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":["risk-monitoring"]},"reading":{"id":"shade-arena","title":"SHADE-Arena: Evaluating sabotage and monitoring in LLM agents","en":{"question":"Can monitors catch sabotage hidden inside otherwise legitimate tool-using tasks?","design":"Self-contained environments give an agent a legitimate task and a covert side task; a separate monitor judges the resulting behavior.","findings":["The best tested monitor, Gemini 2.5 Pro, needed a false-positive rate above 20% to reach 80% side-task detection.","Prompt changes and transcript fine-tuning sometimes improved monitoring (Results)."],"interpretation":"NUWA interpretation: monitoring quality is an operating-point trade-off, not a single detection percentage.","limits":"These are induced side tasks in simulations; the required false-positive rate was judged infeasible for real deployment by the authors.","decisionRelevance":"Model recommendation: retain true/false-positive operating points and distinguish induced sabotage from spontaneous behavior.","takeaway":"NUWA interpretation: monitoring quality is an operating-point trade-off, not a single detection percentage.","connections":[],"scoring":"Side-task detection at a specified false-positive rate, separate from sabotage completion.","conditions":"Synthetic tool environments, induced covert objective and a separate monitor.","statistics":"The best tested monitor, Gemini 2.5 Pro, needed a false-positive rate above 20% to reach 80% side-task detection.","usage":"Model recommendation: retain true/false-positive operating points and distinguish induced sabotage from spontaneous behavior."},"zh":{"question":"监测器能否识别合法工具任务中隐藏的蓄意破坏？","design":"封闭实验环境给智能体合法任务与隐蔽副任务，另一监测模型判断行为。","findings":["最佳受测监测器Gemini 2.5 Pro要达到80%检出，需要超过20%误报。","提示调整与轨迹微调有时可改善监测（结果）。"],"interpretation":"NUWA 解读：监测质量是运行阈值权衡，不是单一检出百分比。","limits":"这是模拟中诱导的副任务；作者认为所需误报率不适合实际部署。","decisionRelevance":"模型建议：保留真阳性、假阳性工作点，并区分诱导与自发破坏。","takeaway":"NUWA 解读：监测质量是运行阈值权衡，不是单一检出百分比。","connections":[],"scoring":"给定误报率的副任务检出，与破坏完成分开。","conditions":"合成工具环境、诱导隐蔽目标及独立监测器。","statistics":"最佳受测监测器Gemini 2.5 Pro要达到80%检出，需要超过20%误报。","usage":"模型建议：保留真阳性、假阳性工作点，并区分诱导与自发破坏。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"shade-arena","url":"https://www.anthropic.com/research/shade-arena-sabotage-monitoring","locator":"SHADE-Arena","version":"Selected original resource https://www.anthropic.com/research/shade-arena-sabotage-monitoring; sha256:0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d; archived source capture; exact capture time not independently established","artifactSha256":"0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d"},{"sourceId":"shade-arena","url":"https://www.anthropic.com/research/shade-arena-sabotage-monitoring","locator":"Results","version":"Selected original resource https://www.anthropic.com/research/shade-arena-sabotage-monitoring; sha256:0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d; archived source capture; exact capture time not independently established","artifactSha256":"0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d"},{"sourceId":"shade-arena","url":"https://www.anthropic.com/research/shade-arena-sabotage-monitoring","locator":"Conclusions","version":"Selected original resource https://www.anthropic.com/research/shade-arena-sabotage-monitoring; sha256:0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d; archived source capture; exact capture time not independently established","artifactSha256":"0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"ba592546fb8e411f0fa61850481c37431bf768a4c0a8f4076ac722b757fc1a55"}},"sources":[{"id":"shade-arena","title":"SHADE-Arena: Evaluating sabotage and monitoring in LLM agents","url":"https://www.anthropic.com/research/shade-arena-sabotage-monitoring","version":"Anthropic research article 2025-06-16","publicationDate":"2025-06-16","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-25","documentId":"https://www.anthropic.com/research/shade-arena-sabotage-monitoring"}]}