NUWA Evaluation dossier
BashArena: A Control Setting for Highly Privileged AI Agents
Provides a control setting for privileged agents, combining legitimate administration tasks with sabotage objectives.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system in the original evaluation configuration
- Measurement target
- Provides a control setting for privileged agents, combining legitimate administration tasks with sabotage objectives.
- Interpretation boundary
- Action-level alerts, trajectory-level false positives and undetected attack success are distinct metrics.
BashArena supplies containerized, privileged system-administration tasks with legitimate main objectives and security-failure side objectives. Honest and adversarial policies are evaluated on task tests, side-task outcomes, monitoring and sandbox restrictions.
Research interpretation
The authors contrast BashArena with the narrower BashBench side task and with APPS backdooring’s single-action setting.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- How well can agents perform useful privileged work, sabotage it, and detect sabotage in a common environment?
- Task population
- Generated administration tasks pair with four fixed side-task families. Monitors inspect action history; attack and monitor comparisons hold different counterpart policies fixed.
- Scoring and failure modes
- Not established by the retained material
- System conditions
- Not established by the retained material
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- Only four sabotage families are included. Attack selection was not explored, elicitation concentrated on one model, and generated tests sometimes reward incomplete solutions.
- Appropriate research use
- NUWA recommendation: retain the honest calibration policy, trajectory false-positive rate and attack families when comparing control protocols.
- BashArena: A Control Setting for Highly Privileged AI Agents · 2 DatasetSelected original resource https://arxiv.org/html/2512.15688; sha256:d492196ad3ecd5463eb9964d4ff5a9095374673e84900d06ad7a76d8305474a0; archived source capture; exact capture time not independently established
- BashArena: A Control Setting for Highly Privileged AI Agents · 4.1 MethodologySelected original resource https://arxiv.org/html/2512.15688; sha256:d492196ad3ecd5463eb9964d4ff5a9095374673e84900d06ad7a76d8305474a0; archived source capture; exact capture time not independently established
- BashArena: A Control Setting for Highly Privileged AI Agents · 4.2 Sabotage resultsSelected original resource https://arxiv.org/html/2512.15688; sha256:d492196ad3ecd5463eb9964d4ff5a9095374673e84900d06ad7a76d8305474a0; archived source capture; exact capture time not independently established
- BashArena: A Control Setting for Highly Privileged AI Agents · 4.3 Monitoring ResultsSelected original resource https://arxiv.org/html/2512.15688; sha256:d492196ad3ecd5463eb9964d4ff5a9095374673e84900d06ad7a76d8305474a0; archived source capture; exact capture time not independently established
- BashArena: A Control Setting for Highly Privileged AI Agents · 5 LimitationsSelected original resource https://arxiv.org/html/2512.15688; sha256:d492196ad3ecd5463eb9964d4ff5a9095374673e84900d06ad7a76d8305474a0; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] Adam Kaufman, James Lucassen. BashArena: A Control Setting for Highly Privileged AI Agents. 2025-12-17. v1 [arXiv]; Author affiliations on the original paper title page; Source figure