{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"metr-monitorability","title":"Early work on monitorability evaluations","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"study","sourceId":"metr-monitorability-publication","attributionSourceId":"metr-monitorability-publication","originalUrl":"https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/","publicationDate":"2026-01-22","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"metr","role":"publisher","sourceId":"metr-monitorability-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Tests monitors against agents attempting side tasks and studies how agents may evade the available oversight.","zh":"让监测器面对尝试额外任务的智能体，研究智能体如何规避既有监督。"},"note":{"question":{"en":"Can an agent complete an instructed side task without a monitor noticing, and how does reasoning access change the outcome?","zh":"智能体能否在不被监测器发现的情况下完成被指派的附加任务，访问推理又如何改变结果？"},"method":{"en":"The SHUSHCAST prototype pairs HCAST main tasks with side tasks. Monitors see the main task and transcript, but not the side-task instruction. Thresholds are calibrated on innocent runs; comparisons vary agent–monitor pairing and access to raw or summarized reasoning.","zh":"SHUSHCAST 原型将 HCAST 主要任务与附加任务配对。监测器看到主要任务和交互记录，但看不到附加任务指令；阈值用正常运行校准，并比较智能体—监测器组合以及原始或摘要推理的访问权限。"},"finding":{"en":"More capable models perform better both as monitors and as agents pursuing side tasks. Reasoning access improves detection substantially for some tested agents, while effects differ for other agents and reasoning formats.","zh":"能力较强的模型在监测和执行附加任务两种角色上均表现更好。访问推理显著改善了部分被测智能体的检测结果，但对其他智能体及推理格式的影响不同。"},"supports":{"en":"The prototype separates successful side-task completion from remaining undetected, allowing the monitored behavior and the monitor’s observation access to be examined together.","zh":"该原型区分附加任务成功与未被检测，使被监测行为与监测器观察权限可以共同分析。"},"limits":{"en":"The source labels this early-stage work: main tasks are few and narrow, and agent/monitor elicitation is limited. Raw and summarized reasoning are not interchangeable, and the task sample does not support a general probability of real-world sabotage evasion.","zh":"来源明确将其视为早期工作：主要任务较少且范围较窄，对智能体与监测器的能力激发也有限。原始推理与推理摘要不能直接等同，任务样本不足以支持现实破坏规避的一般概率。"},"connections":null},"resources":[],"noteStatus":{"question":"verified","method":"verified","finding":"verified","supports":"verified","limits":"verified","connections":"limited"},"noteAvailability":{"en":"The substantive sections shown here were checked against the original source. Unfilled sections are omitted; source checking is not experimental replication.","zh":"此处展示的实质性章节已对照原始来源核查；未完成的章节已省略。来源核查不等于实验复现。"},"noteSourceLocator":"Original METR article (page date 22 January 2026), Methodology; Results; Transcript analysis; Limitations and Future Work; Errata","tags":["risk-monitoring"]},"reading":{"id":"metr-monitorability","title":"Early work on monitorability evaluations","en":{"question":"How well do agent–monitor pairs detect covert side tasks in software work?","design":"SHUSHCAST uses six HCAST main tasks and side tasks; monitor thresholds target 5% false positives on innocent main-task runs, with reasoning visibility varied.","findings":["Agent/monitor capability ratio appeared predictive of uncaught success.","The preliminary study found no convincing relationship between task human duration and ease of monitoring."],"interpretation":"NUWA interpretation: relative monitor strength and reasoning access matter, but task-length trends remain unresolved.","limits":"Six main tasks provide little diversity; agents and monitors were not strongly elicited for this dual-task setting.","decisionRelevance":"Model recommendation: treat trends as preliminary and retain reasoning visibility, 5% FPR threshold and side-task success conditions.","takeaway":"NUWA interpretation: relative monitor strength and reasoning access matter, but task-length trends remain unresolved.","connections":[],"scoring":"Uncaught side-task success at a monitor threshold calibrated to 5% false positives on innocent runs.","conditions":null,"statistics":null,"usage":"Model recommendation: treat trends as preliminary and retain reasoning visibility, 5% FPR threshold and side-task success conditions."},"zh":{"question":"软件工作中，智能体与监测器配对能否检测隐蔽副任务？","design":"SHUSHCAST使用六项HCAST主任务及副任务，在无副任务运行上标定5%误报，并改变推理可见性。","findings":["智能体相对监测器能力似乎预测未被发现的成功。","初步研究未发现人类任务时长与监测难度有可信关系。"],"interpretation":"NUWA 解读：相对强度与推理可见性重要，任务时长趋势仍未解决。","limits":"六项主任务多样性有限，且未充分激发双方在双任务设置的能力。","decisionRelevance":"模型建议：保留初步状态、推理可见性、5%误报阈值及副任务成功条件。","takeaway":"NUWA 解读：相对强度与推理可见性重要，任务时长趋势仍未解决。","connections":[],"scoring":"在无副任务运行校准5%误报阈值下计未检出副任务成功。","conditions":null,"statistics":null,"usage":"模型建议：保留初步状态、推理可见性、5%误报阈值及副任务成功条件。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"metr-monitorability-publication","url":"https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/","locator":"Methodology","version":"Selected original resource https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/; sha256:20f8ce5c450d27d92f70992f203c64f8fd0c03f0fc3a406f20460eca7209beaf; archived source capture; exact capture time not independently established","artifactSha256":"20f8ce5c450d27d92f70992f203c64f8fd0c03f0fc3a406f20460eca7209beaf"},{"sourceId":"metr-monitorability-publication","url":"https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/","locator":"Results","version":"Selected original resource https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/; sha256:20f8ce5c450d27d92f70992f203c64f8fd0c03f0fc3a406f20460eca7209beaf; archived source capture; exact capture time not independently established","artifactSha256":"20f8ce5c450d27d92f70992f203c64f8fd0c03f0fc3a406f20460eca7209beaf"},{"sourceId":"metr-monitorability-publication","url":"https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/","locator":"Limitations and Future Work","version":"Selected original resource https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/; sha256:20f8ce5c450d27d92f70992f203c64f8fd0c03f0fc3a406f20460eca7209beaf; archived source capture; exact capture time not independently established","artifactSha256":"20f8ce5c450d27d92f70992f203c64f8fd0c03f0fc3a406f20460eca7209beaf"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"49b0a664b8ed2c7336f2bd64b73217464772b32178d9755dff1703a5e1213f5c"}},"sources":[{"id":"metr-monitorability-publication","title":"Early work on monitorability evaluations","url":"https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations/","version":"Source capture 2026-10-25","publicationDate":"2026-01-22","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://metr.org/blog/2026-01-19-early-work-on-monitorability-evaluations"}]}