NUWA Risk Evaluation · Benchmark
MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch connectionsResearch and decision relevanceEvidence updateInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
MonitorBench asks which decision-relevant factors become visible to a monitor. A monitorability score is conditional on task construction, monitor access and the scoring rule; it is not a universal guarantee that reasoning is faithful.
Research question
When does a chain of thought reveal factors driving an answer, and how does that visibility change under direct concealment or monitor-aware evasion?
Design and assumptions
The benchmark combines input interventions, outcome justification and verifiable intermediate solution processes. Cue and goal interventions compare nominal and modified inputs. Dual-objective tasks add a side requirement; solution-process tasks require intermediate updates. Monitors receive actions, reasoning or all messages. Stress tests explicitly request concealment or evasion, changing the elicitation condition rather than measuring spontaneous hidden intent.
Source findings
Decision factors structurally embedded in planning tend to be easier to monitor than factors that can change an answer without explicit planning (§4.1).
Reasoning-only monitoring usually carries more information than action-only monitoring in the examined tasks, but visibility varies across tasks and model families (§4.1).
Concealment and evasion reduce visibility for some categories while others remain relatively robust. Failure analysis includes explicit disclosure, mentioning the concealment objective and implicit planning traces (§§4.2–4.3).
NUWA interpretation
The benchmark helps locate where a particular monitor has usable evidence. The observed cross-model capability correlation does not establish that greater capability causally reduces visibility. Detecting an injected side objective is different from detecting naturally arising harmful intent.
Limits and contrary evidence
Task designs create known decision factors; they do not cover all deployment failures. The input-intervention score depends on a meaningful treatment effect and the source’s attribution assumptions. Capping at one hides excess flagging from that endpoint, so calibration matters. Stress-test instructions and monitored message scope must accompany any result.
Research connections
Alignment Faking relies partly on scratchpad-based reasoning classification. AI Control evaluates a complete defense against an adaptive attack policy; MonitorBench alone does not measure whether a defense prevents harm.
Research and decision relevance
Model recommendation: select monitor access and a tolerable false-positive regime before comparison, preserve nominal controls, and test monitoring inside a defense protocol rather than treating visibility as safety.
Precise source locations
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.1 Evaluation Design; Table 1arXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.2.1 Input InterventionarXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.2.2 Outcome JustificationarXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.2.3 Solution ProcessarXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.3 Evaluation MetricsarXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §4.1 Monitorability Across Factors and Model Families; Figure 2arXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §4.2 Monitorability Under Stress-Test; Table 3arXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §4.3 Mechanisms of Failure and Evasion; Table 2arXiv:2603.28590v2
- MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · Appendix A.4.2 Monitor CalibrationarXiv:2603.28590v2
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
Separate recognition on fixed traces from safety diagnosis during execution. Self-assessment recall depends on its teacher labels; TraceProbe replay localises recorded actions rather than rerunning the model in a counterfactual.
Related reading
NUWA reading connections; these do not imply citation, replication or score equivalence.
Sources for this interpretation
- Scaling Misuse Detection with Self-Assessment · Results; limitations; text lines 00008,00014,00060Original resource captured 2026-10-28T07:24:41.534870+00:00; sha256:03b9ece6dae1c006622bb47373fb914bec20cee054a452748f30573c63ff6110
- TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents · §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483Original resource captured 2026-10-28T07:24:41.534922+00:00; sha256:b0b85491f4be9fb842c2f97891b033a75b44a85c756b0f3b92cf8da775d8f828
Institutions and roles
Authors (11)
Han Wang · Yifan Sun · Brian Ko · Mann Talati · Jiawen Gong · Zimeng Li · Naicheng Yu · Xucheng Yu · Wei Shen · Vedant Jolly · Huan Zhang
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{monitorbench,
title = {MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models},
url = {https://arxiv.org/abs/2603.28590v2},
author = {Han Wang and Yifan Sun and Brian Ko and Mann Talati and Jiawen Gong and Zimeng Li and Naicheng Yu and Xucheng Yu and Wei Shen and Vedant Jolly and Huan Zhang},
year = {2026},
date = {2026-03-30},
version = {arXiv:2603.28590v2},
note = {arXiv:2603.28590v2}
}
NUWA record · BibTeX
@misc{monitorbench-nuwa,
title = {MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/monitorbench/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--monitorbench.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--monitorbench.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] Han Wang, Yifan Sun, et al.. MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models. 2026-03-30. v2 [arXiv]; Author affiliations on the original paper title page; §3.1 Evaluation Design; Table 1; §3.2.1 Input Intervention