NUWA Evaluation dossier
SHADE-Arena
Pairs benign multi-step tasks with explicitly assigned covert side tasks and monitoring.
Task and measurement
Benchmark protocol profile
- Task / subject
- Agents assigned a covert side task alongside a legitimate task
- Measurement target
- Covert side-task completion and monitor detection under defined tool access.
- Interpretation boundary
- An assigned malicious side task measures ability to execute it, not spontaneous desire to sabotage.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can monitors catch sabotage hidden inside otherwise legitimate tool-using tasks?
- Task population
- Self-contained environments give an agent a legitimate task and a covert side task; a separate monitor judges the resulting behavior.
- Scoring and failure modes
- Side-task detection at a specified false-positive rate, separate from sabotage completion.
- System conditions
- Synthetic tool environments, induced covert objective and a separate monitor.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: The best tested monitor, Gemini 2.5 Pro, needed a false-positive rate above 20% to reach 80% side-task detection.
- Validity and interpretation
- These are induced side tasks in simulations; the required false-positive rate was judged infeasible for real deployment by the authors.
- Appropriate research use
- Model recommendation: retain true/false-positive operating points and distinguish induced sabotage from spontaneous behavior.
- SHADE-Arena: Evaluating sabotage and monitoring in LLM agents · SHADE-ArenaSelected original resource https://www.anthropic.com/research/shade-arena-sabotage-monitoring; sha256:0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d; archived source capture; exact capture time not independently established
- SHADE-Arena: Evaluating sabotage and monitoring in LLM agents · ResultsSelected original resource https://www.anthropic.com/research/shade-arena-sabotage-monitoring; sha256:0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d; archived source capture; exact capture time not independently established
- SHADE-Arena: Evaluating sabotage and monitoring in LLM agents · ConclusionsSelected original resource https://www.anthropic.com/research/shade-arena-sabotage-monitoring; sha256:0a989a6b3149554677b8b01058a0d2b97da95a26b0063898bfa438824d8ed08d; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
Deception / Covert goal pursuit →
The protocol tests whether agents can complete a covert objective while doing legitimate work.
References
- [1] SHADE-Arena: Evaluating sabotage and monitoring in LLM agents. 2025-06-16 ; Official publication masthead and article byline
- [2] SHADE-Arena: Evaluating sabotage and monitoring in LLM agents. 2025-06-16 ; Figure 3