NUWA RESEARCH RECORD
Simulated Control Evaluations
Original publication ↗ · BibTeX · RISSimulation predicts pooled safety more accurately than the simpler BashArena comparison.Real-world transcript checks do not validate attack success because those sessions contain no attacks.At5% audit budget, pooled safety-prediction RMSE is3.6 percentage points for grounded simulation versus10.1 for BashArena; errors are squared within each monitor before pooling.Simulation explains95% of expected-safety variation between side-task categories and78% between environments. It matches the BashArena prediction using4.6% as many task-pair draws.
Research question
Can simulated environments predict control results in executable environments?
Design & assumptions
Compares simulated and executable LinuxArena trajectories and protocol metrics.
Findings
Simulation predicts pooled safety more accurately than the simpler BashArena comparison.
Real-world transcript checks do not validate attack success because those sessions contain no attacks.
At5% audit budget, pooled safety-prediction RMSE is3.6 percentage points for grounded simulation versus10.1 for BashArena; errors are squared within each monitor before pooling.
Simulation explains95% of expected-safety variation between side-task categories and78% between environments. It matches the BashArena prediction using4.6% as many task-pair draws.
Limits & counterevidence
Unseen-environment transfer, untrusted simulators and multi-agent attacks remain unresolved.
Source evidence
| System | Measure | Reported value | Conditions & location |
|---|---|---|---|
| Grounded simulator | Pooled safety prediction RMSE | 3.6 percentage pointspercentage points | LinuxArena execution reference; 5% audit budget for pooled RMSE; Grounded LinuxArena simulatorL87-L92 |
| BashArena | Pooled safety prediction RMSE | 10.1percentage points | LinuxArena execution reference; 5% audit budget for pooled RMSE; BashArena comparisonL87-L92 |
| Grounded simulator | Explained variation across side-task categories | 95%% | LinuxArena execution reference; 5% audit budget for pooled RMSE; Expected safety across side-task categoriesL87-L92 |
| Grounded simulator | Explained variation across environments | 78%% | LinuxArena execution reference; 5% audit budget for pooled RMSE; Expected safety across LinuxArena environmentsL87-L92 |
| Grounded simulator | Category safety standard deviation | 20.2 percentage pointspercentage points | LinuxArena execution reference; 5% audit budget for pooled RMSE; Between side-task categoriesL87-L92 |
| Grounded simulator | Simulation RMS bias | 4.6 pointspercentage points | LinuxArena execution reference; 5% audit budget for pooled RMSE; Expected error excluding rollout varianceL87-L92 |
| Grounded simulator | Task-pair draws needed to beat comparison | 4.6%% | LinuxArena execution reference; 5% audit budget for pooled RMSE; Fraction of deployment task-pair draws needed to beat full BashArena safety predictionL87-L92 |
Reading scope & review
Selected substantive passages of original institutional article; linked full paper, data and figures not independently audited
Model-assisted reading; independent evidence review pending.
Check the original source ↗