NUWA Evidence record
GPT-5.2
BashArena and Control Setting Design · Evidence record
Original source charts and materials
Recorded measurements
- Main task usefulness
- 0.66 fraction of test cases passed
Not disclosed
BashArena · Budget not specified
Method and scope
What makes a control setting informative about deployment safeguards?
- Evidence type
- Benchmark
- Assessment subject
- Pairs system-administration main tasks with scored security side tasks.
- Primary dimension
- Behavior
- Evidence snapshot
- 2026-10-29.1
Control-game usefulness is not an absolute capability ranking; model evaluations use different task subsets.
Source finding
Task generation preserves a capability gap between trusted and untrusted models. The authors document grading weaknesses and overpowered strategies requiring correction. At 4% trajectory-wise false positives, the GPT-4.1 mini monitor reaches 74% safety. The associated 0.08% stepwise FPR uses a different denominator. Removing internet access changes partial-credit usefulness from 59% to 55% while disabling three of four side tasks; this exposes a setting-design shortcut. Honest Sonnet4.5 and GPT-4.1 mini policies each accidentally trigger the firewall task once in637 trajectories; no other side tasks trigger accidentally.
Limitations
Control-game usefulness is not an absolute capability ranking; model evaluations use different task subsets.
Control-game usefulness is not an absolute capability ranking; model evaluations use different task subsets.
Related research questions
References
Numerical results are reported by the cited authors.