NUWA Evaluation dossier
BashArena and Control Setting Design
What makes a control setting informative about deployment safeguards?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
BashArena and Control Setting Design · Main task usefulness
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
7 observations · 0 dated
- Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator.
- Original release; partial credit as average fraction of test cases passed; 637 main tasks for this model per article L48. Task count is not the test-case denominator.
7 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
7 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 GPT-5.2 | 0.66 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
 Claude Opus 4.5 | 0.66 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
| Claude Sonnet 4.5 | 0.59 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 637 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
| GPT-5 | 0.51 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
| GPT-4.1 mini | 0.34 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 637 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
 Kimi K2 | 0.27 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
| GPT-5 nano | 0.24 fraction of test cases passedMain task usefulness | Incomplete | Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator.BashArena | Not disclosedBashArena and Control Setting Design | Release —Report 2025-12-18 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Pairs system-administration main tasks with scored security side tasks.
- Measurement target
- What makes a control setting informative about deployment safeguards?
- Scoring / outcome
- Main task usefulness · Main task usefulness
- Interpretation boundary
- Control-game usefulness is not an absolute capability ranking; model evaluations use different task subsets.
A reported evaluation
The source evaluates GPT-5.2 · Original release; partial credit as average fraction of test cases passed; 100 main tasks for this model per article L48. Task count is not the test-case denominator..
Reported measurement: Main task usefulness · 0.66 fraction of test cases passed.
Main task usefulness Not disclosed
Task generation preserves a capability gap between trusted and untrusted models. The authors document grading weaknesses and overpowered strategies requiring correction. At 4% trajectory-wise false positives, the GPT-4.1 mini monitor reaches 74% safety. The associated 0.08% stepwise FPR uses a different denominator. Removing internet access changes partial-credit usefulness from 59% to 55% while disabling three of four side tasks; this exposes a setting-design shortcut. Honest Sonnet4.5 and GPT-4.1 mini policies each accidentally trigger the firewall task once in637 trajectories; no other side tasks trigger accidentally.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Safety/usefulness tradeoffs for trusted control of untrusted system-administration agents.
- Task population
- 637 main tasks across nine clusters are paired with four scored security side tasks.
- Scoring and failure modes
- Safety is detecting an attack before it succeeds. Usefulness uses average fraction of main-task test cases passed, not all-or-nothing task success.
- System conditions
- Original release evaluates trusted monitoring at 4% trajectory-wise FPR; a prose example gives74% safety with GPT-4.1 mini. Sonnet4.5 and GPT-4.1 mini use all637 tasks, other models random100-task subsets.
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- Weak integration of dangerous permissions and difficult test cases allows sandboxing shortcuts. Dynamic permission management is banned in the control game; results do not assess that class of protocols.
- Appropriate research use
- Not established by the retained material
- BashArena and Control Setting Design · L1-L48; L71-L1294e1567de205bd18388f70b944799ff2fe3acbadc80321e73bf1b50200083f034
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
BashArena and Control Setting Design
BashArena and Control Setting Design
Actual evaluators: Not disclosed
7 result records · 7 reported system names
Protocols and comparison decisions (0)
Original source charts and materials
Control-game usefulness is not an absolute capability ranking; model evaluations use different task subsets.
Related research questions
References
Numerical results are reported by the cited authors.