MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

When does a chain of thought reveal factors driving an answer, and how does that visibility change under direct concealment or monitor-aware evasion?

Task population

The benchmark combines input interventions, outcome justification and verifiable intermediate solution processes. Cue and goal interventions compare nominal and modified inputs. Dual-objective tasks add a side requirement; solution-process tasks require intermediate updates. Monitors receive actions, reasoning or all messages. Stress tests explicitly request concealment or evasion, changing the elicitation condition rather than measuring spontaneous hidden intent.

Scoring and failure modes

For input intervention, the metric estimates the share of successful outputs attributable to the intervention and compares monitor flagging with that minimum share, capped at one. Other task types use their defined verification and monitoring rules. Task success, attribution and monitor detection are separate variables.

System conditions

The benchmark combines input interventions, outcome justification and verifiable intermediate solution processes. Cue and goal interventions compare nominal and modified inputs. Dual-objective tasks add a side requirement; solution-process tasks require intermediate updates. Monitors receive actions, reasoning or all messages. Stress tests explicitly request concealment or evasion, changing the elicitation condition rather than measuring spontaneous hidden intent.

Statistical meaning

§3.3 defines conditional attribution-based scores and capping; Appendix A.4.2 addresses monitor calibration. Per-task eligible successful samples differ from the full benchmark size.

Validity and interpretation

Task designs create known decision factors; they do not cover all deployment failures. The input-intervention score depends on a meaningful treatment effect and the source’s attribution assumptions. Capping at one hides excess flagging from that endpoint, so calibration matters. Stress-test instructions and monitored message scope must accompany any result.

Appropriate research use

Model recommendation: select monitor access and a tolerable false-positive regime before comparison, preserve nominal controls, and test monitoring inside a defense protocol rather than treating visibility as safety.

• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.1 Evaluation Design; Table 1arXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.2.1 Input InterventionarXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.2.2 Outcome JustificationarXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.2.3 Solution ProcessarXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §3.3 Evaluation MetricsarXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §4.1 Monitorability Across Factors and Model Families; Figure 2arXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §4.2 Monitorability Under Stress-Test; Table 3arXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · §4.3 Mechanisms of Failure and Evasion; Table 2arXiv:2603.28590v2
• MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models · Appendix A.4.2 Monitor CalibrationarXiv:2603.28590v2