{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"monitorbench","title":"MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models","aliases":[],"authors":["Han Wang","Yifan Sun","Brian Ko","Mann Talati","Jiawen Gong","Zimeng Li","Naicheng Yu","Xucheng Yu","Wei Shen","Vedant Jolly","Huan Zhang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"monitorbench-publication","attributionSourceId":"monitorbench-publication-author-source","originalUrl":"https://arxiv.org/abs/2603.28590v2","publicationDate":"2026-03-30","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":["monitorbench"],"contributions":[{"organizationId":"uiuc","role":"research","sourceId":"monitorbench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"washington","role":"research","sourceId":"monitorbench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"ucsd","role":"research","sourceId":"monitorbench-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Benchmarks chain-of-thought monitorability across tasks that include hidden objectives, underperformance and encoded communication.","zh":"通过隐藏目标、低表现及编码通信等任务评测推理链可监测性。"},"note":{"question":{"en":"When do decision-relevant factors appear in a model’s chain of thought, and how much can concealment instructions reduce their detectability?","zh":"决策相关因素何时会出现在模型思维链中，隐瞒指令又能在多大程度上降低其可检测性？"},"method":{"en":"MonitorBench defines decision-critical factors across input-intervention, outcome-justification and solution-process tasks. Standard evaluation is paired with direct-concealment and monitor-aware-evasion stress tests, comparing models and task categories.","zh":"MonitorBench 在输入干预、结果解释和解题过程任务中定义决策关键因素，将常规评估与直接隐瞒、感知监测器的规避压力测试配对，并比较模型和任务类别。"},"finding":{"en":"The study reports stronger monitorability when a factor shapes intermediate reasoning rather than only the final response. Stress tests reduce monitorability on some tasks, often through omission of the factor rather than complicated obfuscation.","zh":"研究报告，当某因素影响中间推理而不仅是最终回答时，可监测性更强。压力测试降低了部分任务的可监测性，其方式往往是省略该因素，而非复杂混淆。"},"supports":{"en":"The benchmark helps distinguish types of monitoring targets and tests deliberate concealment. Its aggregate patterns should be read alongside task-specific conditions.","zh":"该基准有助于区分监测目标类型并测试主动隐瞒；总体模式仍需结合具体任务条件理解。"},"limits":{"en":"These are constructed decision-factor tasks and specified stress-test prompts. Reduced verbalization does not by itself establish an independent malicious objective, and cross-model patterns are not a causal test of capability growth.","zh":"这些结果来自构造的决策因素任务与指定压力测试提示。减少表述本身不能证明存在独立恶意目标，跨模型模式也不是能力增长的因果检验。"},"connections":null},"resources":[],"noteStatus":{"question":"verified","method":"verified","finding":"verified","supports":"verified","limits":"verified","connections":"limited"},"noteAvailability":{"en":"The substantive sections shown here were checked against the original source. Unfilled sections are omitted; source checking is not experimental replication.","zh":"此处展示的实质性章节已对照原始来源核查；未完成的章节已省略。来源核查不等于实验复现。"},"noteSourceLocator":"arXiv:2603.28590v2, §§3.1–3.3, 4.1–4.3; Appendix A.3–A.4","tags":["risk-monitoring"]},"reading":{"id":"monitorbench","title":"MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models","en":{"takeaway":"MonitorBench asks which decision-relevant factors become visible to a monitor. A monitorability score is conditional on task construction, monitor access and the scoring rule; it is not a universal guarantee that reasoning is faithful.","question":"When does a chain of thought reveal factors driving an answer, and how does that visibility change under direct concealment or monitor-aware evasion?","design":"The benchmark combines input interventions, outcome justification and verifiable intermediate solution processes. Cue and goal interventions compare nominal and modified inputs. Dual-objective tasks add a side requirement; solution-process tasks require intermediate updates. Monitors receive actions, reasoning or all messages. Stress tests explicitly request concealment or evasion, changing the elicitation condition rather than measuring spontaneous hidden intent.","scoring":"For input intervention, the metric estimates the share of successful outputs attributable to the intervention and compares monitor flagging with that minimum share, capped at one. Other task types use their defined verification and monitoring rules. Task success, attribution and monitor detection are separate variables.","findings":["Decision factors structurally embedded in planning tend to be easier to monitor than factors that can change an answer without explicit planning (§4.1).","Reasoning-only monitoring usually carries more information than action-only monitoring in the examined tasks, but visibility varies across tasks and model families (§4.1).","Concealment and evasion reduce visibility for some categories while others remain relatively robust. Failure analysis includes explicit disclosure, mentioning the concealment objective and implicit planning traces (§§4.2–4.3)."],"interpretation":"The benchmark helps locate where a particular monitor has usable evidence. The observed cross-model capability correlation does not establish that greater capability causally reduces visibility. Detecting an injected side objective is different from detecting naturally arising harmful intent.","limits":"Task designs create known decision factors; they do not cover all deployment failures. The input-intervention score depends on a meaningful treatment effect and the source’s attribution assumptions. Capping at one hides excess flagging from that endpoint, so calibration matters. Stress-test instructions and monitored message scope must accompany any result.","connections":["Alignment Faking relies partly on scratchpad-based reasoning classification. AI Control evaluates a complete defense against an adaptive attack policy; MonitorBench alone does not measure whether a defense prevents harm."],"decisionRelevance":"Model recommendation: select monitor access and a tolerable false-positive regime before comparison, preserve nominal controls, and test monitoring inside a defense protocol rather than treating visibility as safety.","statistics":"§3.3 defines conditional attribution-based scores and capping; Appendix A.4.2 addresses monitor calibration. Per-task eligible successful samples differ from the full benchmark size."},"zh":{"takeaway":"MonitorBench 检查哪些决策因素对监测者可见；分数依赖任务构造、可见消息和评分规则，不是推理普遍忠实的保证。","question":"思维链何时揭示决定答案的因素？直接隐瞒或知晓监测后的规避会如何改变可见性？","design":"基准结合输入干预、结果解释与可验证中间步骤。线索和目标干预比较原始与修改输入；双目标任务增加侧面要求，解题过程任务要求连续更新。监测者分别看到行动、推理或全部消息；压力测试明确要求隐瞒或规避，改变的是引出条件，并非自发隐藏意图的测量。","scoring":"输入干预指标先估计成功输出中归因于干预的最低比例，再比较监测标记与该比例，并封顶为一；其他类别按各自验收与监测规则。任务成功、归因和检测是独立变量。","findings":["与规划结构紧密相关的因素通常较易监测，只改变答案而无需显式规划的因素较难（§4.1）。","所研究任务中，仅推理监测通常比仅行动监测提供更多信息，但可见性随任务和模型族变化（§4.1）。","隐瞒和规避降低部分类别的可见性，另一些类别较稳定；失败分析包含直接披露、提及隐瞒目标与隐式规划痕迹（§§4.2–4.3）。"],"interpretation":"基准帮助定位监测器可用证据，但跨模型的能力相关性不证明能力提升因果性降低可见性；检测注入侧目标也不同于检测自然出现的有害意图。","limits":"设计提供已知决策因素，不覆盖所有部署失败。输入干预指标依赖有意义的处理效应及归因假设，封顶使额外标记不能体现在该终点中，因此需核查校准。压力提示和消息可见范围必须随结果保留。","connections":["伪装对齐部分依赖草稿推理分类；AI Control 则评估面对适应性攻击策略的完整防御，MonitorBench 单独不能测量防御是否阻止伤害。"],"decisionRelevance":"模型建议：先选择消息可见范围和可接受假阳性条件，保留原始输入对照，并在完整防御协议中测试监测。","statistics":"§3.3 定义条件归因分数和封顶；附录 A.4.2 讨论校准。各任务合格成功样本的分母不同于基准总实例数。"},"figureRefs":[{"figureId":"monitorbench-figure","sourceId":"monitorbench-figure-source","locator":"Figure 2","original":"/images/nuwa-index/monitorbench-original.png","provenanceUrl":"https://arxiv.org/html/2603.28590v2/Fig/combined_v14_open_closed_avg.png","disposition":"existing-repository-provenance-not-newly-authorized","review":"Retain existing repository provenance only. Image suitability and rights are not newly approved by this editorial draft."}],"assertionPolicy":{"design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","connections":"nuwa-interpretation-with-source-refs","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§3.1 Evaluation Design; Table 1","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§3.2.1 Input Intervention","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§3.2.2 Outcome Justification","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§3.2.3 Solution Process","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§3.3 Evaluation Metrics","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§4.1 Monitorability Across Factors and Model Families; Figure 2","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§4.2 Monitorability Under Stress-Test; Table 3","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"§4.3 Mechanisms of Failure and Evasion; Table 2","version":"arXiv:2603.28590v2"},{"sourceId":"monitorbench-publication-author-source","url":"https://arxiv.org/html/2603.28590v2","locator":"Appendix A.4.2 Monitor Calibration","version":"arXiv:2603.28590v2"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":"full-text","readingCoverage":"selected-methods-results-limitations-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"6e00e536b6263b98935737d5e54dbc4cc58e98fe83ebacc277314d02812b5d5a"}},"sources":[{"id":"monitorbench-publication","title":"MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models","url":"https://arxiv.org/abs/2603.28590v2","version":"arXiv:2603.28590v2","publicationDate":"2026-03-30","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2603.28590","authors":["Han Wang","Yifan Sun","Brian Ko","Mann Talati","Jiawen Gong","Zimeng Li   Naicheng Yu   Xucheng Yu   Wei Shen   Vedant Jolly","Huan Zhang"],"aliases":["https://doi.org/10.48550/arXiv.2603.28590","https://arxiv.org/abs/2603.28590","https://arxiv.org/pdf/2603.28590"]}]}