NUWA RESEARCH RECORD
Screwtape: Adversarial Transcript Editing for Monitor Red-Teaming
Original publication ↗ · BibTeX · RISA small subset of discovered evasions transferred and passed human validity review.The method tests monitor robustness rather than an agent’s unaided attack propensity.
Research question
How can realistic monitor failures be found without executing attacks?
Design & assumptions
Edits recorded trajectories, uses a privileged verifier and tests transfer to a held-out monitor.
Findings
A small subset of discovered evasions transferred and passed human validity review.
The method tests monitor robustness rather than an agent’s unaided attack propensity.
Limits & counterevidence
No live environment execution; verifier false acceptance requires manual review.
Source evidence
| System | Measure | Reported value | Conditions & location |
|---|---|---|---|
| — | Pipeline runs | 252count | 252 runs on 94 unique transcriptsL4,L61 |
| — | Unique source transcripts | 94count | Source corpus, not independent pipeline runsL4,L61 |
| — | proxy evasions | 35attacks | Attacks evading proxy monitors; candidate setL4 |
| — | validated transfer | Sixattacksn = 35 | Held-out monitor evasion plus human validity review; denominator 35 selected proxy evasionsL4 |
| — | Attacks evading proxy monitors | 35countn = 252 | Proxy monitors based on Opus 4.8 and Gemini 3.1 Pro; selected output of 252 runsL4,L61 |
| — | Attacks evading held-out monitor before review | 17countn = 35 | Final review confirmed 6 of these 17; do not treat all as validL4,L61 |
| — | Confirmed attacks evading held-out monitor | 6countn = 17 | Confirmed by original authors; 6 of 17 reviewed held-out evaders; entire search started with252runsL4,L61 |
Reading scope & review
Selected substantive passages of original institutional article; linked full paper, data and figures not independently audited
Model-assisted reading; independent evidence review pending.
Check the original source ↗