NUWA RESEARCH RECORD
Sabotage evaluations for frontier models
Original publication ↗ · BibTeX · RISAggressive decision sabotage increased errors and suspicion; stronger code reviewers reduced attacker success.
Research question
Can models undermine decisions, code and oversight?
Design & assumptions
Four adversarial evaluations, including a human business-decision experiment and attacker–defender code review.
Findings
Aggressive decision sabotage increased errors and suspicion; stronger code reviewers reduced attacker success.
Limits & counterevidence
Small artificial scenarios measure elicited capability, not deployment prevalence.
Source evidence
| System | Measure | Reported value | Conditions & location |
|---|---|---|---|
| — | sandbag-trained-relative-performance | 36.1%percent-of-normal-performance | Relative to normal performance, not a trial success percentage.Sandbagging |
| — | post-safety-training-relative-performance | 95.6%percent-of-normal-performance | After safety training; original task denominator not disclosed in selected blog.Sandbagging |
| Claude 3.5 Sonnet always-helpful variant | human-suspiciousness-score | 2.65 (±0.25)scoren = 54 participants | control prompt; human business-decision study.PDF p.10, Table 3, control, suspiciousness |
| Claude 3.5 Sonnet always-helpful variant | correct-business-decision-rate | 81% (±5%)%n = 54 participants | control prompt; human business-decision study.PDF p.10, Table 3, control, correct proportion |
| Claude 3.5 Sonnet always-helpful variant | human-suspiciousness-score | 3.17 (±0.44)scoren = 18 participants | subtle prompt; human business-decision study.PDF p.10, Table 3, subtle, suspiciousness |
| Claude 3.5 Sonnet always-helpful variant | correct-business-decision-rate | 67% (±11%)%n = 18 participants | subtle prompt; human business-decision study.PDF p.10, Table 3, subtle, correct proportion |
| Claude 3.5 Sonnet always-helpful variant | human-suspiciousness-score | 4.44 (±0.49)scoren = 18 participants | moderate prompt; human business-decision study.PDF p.10, Table 3, moderate, suspiciousness |
| Claude 3.5 Sonnet always-helpful variant | correct-business-decision-rate | 77% (±10%)%n = 18 participants | moderate prompt; human business-decision study.PDF p.10, Table 3, moderate, correct proportion |
| Claude 3.5 Sonnet always-helpful variant | human-suspiciousness-score | 3.22 (±0.52)scoren = 18 participants | aggressive prompt; human business-decision study.PDF p.10, Table 3, aggressive, suspiciousness |
| Claude 3.5 Sonnet always-helpful variant | correct-business-decision-rate | 44% (±12%)%n = 18 participants | aggressive prompt; human business-decision study.PDF p.10, Table 3, aggressive, correct proportion |
| Claude 3.5 Sonnet always-helpful variant | samples-to-catch-25pct | 17samples | subtle; derived threshold sample estimate, not measured run count.PDF p.11, Table 4, subtle, samples-to-catch-25pct |
| Claude 3.5 Sonnet always-helpful variant | samples-to-sabotage-90pct | 85samples | subtle; derived threshold sample estimate, not measured run count.PDF p.11, Table 4, subtle, samples-to-sabotage-90pct |
| Claude 3.5 Sonnet always-helpful variant | samples-to-catch-25pct | 2samples | moderate; derived threshold sample estimate, not measured run count.PDF p.11, Table 4, moderate, samples-to-catch-25pct |
| Claude 3.5 Sonnet always-helpful variant | samples-to-sabotage-90pct | 1210samples | moderate; derived threshold sample estimate, not measured run count.PDF p.11, Table 4, moderate, samples-to-sabotage-90pct |
| Claude 3.5 Sonnet always-helpful variant | samples-to-catch-25pct | 12samples | aggressive; derived threshold sample estimate, not measured run count.PDF p.11, Table 4, aggressive, samples-to-catch-25pct |
| Claude 3.5 Sonnet always-helpful variant | samples-to-sabotage-90pct | 17samples | aggressive; derived threshold sample estimate, not measured run count.PDF p.11, Table 4, aggressive, samples-to-sabotage-90pct |
| Claude 3.5 Sonnet attacker | defender-firing-rate | 50%%n = 128 minigames | Attack; defender Claude 3 Opus; four-commit minigames; 128 simulations per arm.PDF p.14, Table 5, Attack, Claude 3 Opus |
| Claude 3.5 Sonnet attacker | attacker-win-rate | 26.6%%n = 128 minigames | Attack; defender Claude 3 Opus; high-impact bug and not detected; 128 simulations per arm.PDF p.14, Table 5, Attack, Claude 3 Opus |
| Claude 3.5 Sonnet attacker | defender-firing-rate | 75%%n = 128 minigames | Attack; defender Claude 3.5 Sonnet; four-commit minigames; 128 simulations per arm.PDF p.14, Table 5, Attack, Claude 3.5 Sonnet |
| Claude 3.5 Sonnet attacker | attacker-win-rate | 7.8%%n = 128 minigames | Attack; defender Claude 3.5 Sonnet; high-impact bug and not detected; 128 simulations per arm.PDF p.14, Table 5, Attack, Claude 3.5 Sonnet |
| Claude 3.5 Sonnet attacker | defender-firing-rate | 6.3%%n = 128 minigames | Control; defender Claude 3 Opus; four-commit minigames; 128 simulations per arm.PDF p.14, Table 5, Control, Claude 3 Opus |
| Claude 3.5 Sonnet attacker | defender-firing-rate | 27.3%%n = 128 minigames | Control; defender Claude 3.5 Sonnet; four-commit minigames; 128 simulations per arm.PDF p.14, Table 5, Control, Claude 3.5 Sonnet |
| Claude 3.5 Sonnet attacker | full-game-attacker-win-rate | 11.4%% | Full deployment simulation; proxy Claude 3.5 Sonnet; defender Claude 3 Opus; optimizer BayesOpt. Distinct from four-commit minigame results.PDF p.18, Table 7, Claude 3.5 Sonnet/Claude 3 Opus/BayesOpt |
| Claude 3.5 Sonnet attacker | full-game-attacker-win-rate | 10.1%% | Full deployment simulation; proxy Claude 3.5 Sonnet; defender Claude 3 Opus; optimizer AttackerOpt. Distinct from four-commit minigame results.PDF p.18, Table 7, Claude 3.5 Sonnet/Claude 3 Opus/AttackerOpt |
| Claude 3.5 Sonnet attacker | full-game-attacker-win-rate | 4.3%% | Full deployment simulation; proxy Claude 3 Opus; defender Claude 3.5 Sonnet; optimizer BayesOpt. Distinct from four-commit minigame results.PDF p.18, Table 7, Claude 3 Opus/Claude 3.5 Sonnet/BayesOpt |
| Claude 3.5 Sonnet attacker | full-game-attacker-win-rate | 1.4%% | Full deployment simulation; proxy Claude 3 Opus; defender Claude 3.5 Sonnet; optimizer AttackerOpt. Distinct from four-commit minigame results.PDF p.18, Table 7, Claude 3 Opus/Claude 3.5 Sonnet/AttackerOpt |
| Claude 3.5 Sonnet always-helpful variant | predicted-monitor-correlation | 0.606correlation | Few-shot generalizationPDF p.67, Table 10, Few-shot generalization |
| Claude 3.5 Sonnet always-helpful variant | predicted-monitor-correlation | 0.547correlation | Few-shot i.i.dPDF p.67, Table 10, Few-shot i.i.d |
| Claude 3.5 Sonnet always-helpful variant | predicted-monitor-correlation | 0.497correlation | Finetune generalizationPDF p.67, Table 10, Finetune generalization |
| Claude 3.5 Sonnet always-helpful variant | predicted-monitor-correlation | 0.522correlation | Finetune i.i.dPDF p.67, Table 10, Finetune i.i.d |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 7.73score | Zero-shot; scheming; only top 10% most suspicious evaluations.PDF p.68, Table 11, Zero-shot, scheming |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 2.04score | Zero-shot; baseline; only top 10% most suspicious evaluations.PDF p.68, Table 11, Zero-shot, baseline |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 16.71score | Zero-shot with CoT; scheming; only top 10% most suspicious evaluations.PDF p.68, Table 11, Zero-shot with CoT, scheming |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 15.31score | Zero-shot with CoT; baseline; only top 10% most suspicious evaluations.PDF p.68, Table 11, Zero-shot with CoT, baseline |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 16.85score | Few-shot with CoT; scheming; only top 10% most suspicious evaluations.PDF p.68, Table 11, Few-shot with CoT, scheming |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 15.61score | Few-shot with CoT; baseline; only top 10% most suspicious evaluations.PDF p.68, Table 11, Few-shot with CoT, baseline |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 14.17score | Zero-shot with CoT, no reason; scheming; only top 10% most suspicious evaluations.PDF p.68, Table 11, Zero-shot with CoT, no reason, scheming |
| Claude 3.5 Sonnet always-helpful variant | mean-suspiciousness-top-decile | 13.51score | Zero-shot with CoT, no reason; baseline; only top 10% most suspicious evaluations.PDF p.68, Table 11, Zero-shot with CoT, no reason, baseline |
Reading scope & review
Selected original webpage sections; linked paper, appendix and data not fully audited.
Model-assisted reading; independent evidence review pending.
Check the original source ↗