Evaluation dossier

Stealth evaluations

Reasoning prerequisites for evading oversight.

Google DeepMind / arXiv:2505.01420v4 / Capability

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

Stealth evaluations · Stealth task score

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

16 observations · 12 dated

● Matched◇ Configuration differs□ Some conditions undisclosed

4 observations lack a verified model release date. They remain in Data and Systems.

Select a point or table row to inspect its conditions and provenance.

16 observations

Measurements and source conditions
System / model	Result	Condition status	Task / conditions	Evaluator / report	Dates
Kimi K3	100%Stealth task score	Incomplete	GDM stealthOversight pattern—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-16Report 2026-09-11
DeepSeek V4 FlashDeepSeek V4 Flash 0731	100%Stealth task score	Incomplete	GDM stealthOversight pattern—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release —Report 2026-09-11
Claude Opus 4.6	13%Stealth task score	Incomplete	GDM stealthOversight pattern—	MetaEvaluating Kimi K3 for Misalignment Risks	Release 2026-02-05Report 2026-09-11
GPT-5.6 Terra	0%Stealth task score	Incomplete	GDM stealthOversight pattern—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-09Report 2026-09-11
Kimi K3	78%Stealth task score	Incomplete	GDM stealthClassifier evasion—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-16Report 2026-09-11
DeepSeek V4 FlashDeepSeek V4 Flash 0731	22%Stealth task score	Incomplete	GDM stealthClassifier evasion—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release —Report 2026-09-11
GPT-5.6 Terra	0%Stealth task score	Incomplete	GDM stealthClassifier evasion—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-09Report 2026-09-11
Claude Opus 4.6	0%Stealth task score	Incomplete	GDM stealthClassifier evasion—	MetaEvaluating Kimi K3 for Misalignment Risks	Release 2026-02-05Report 2026-09-11
Kimi K3	100%Stealth task score	Incomplete	GDM stealthCover your tracks—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-16Report 2026-09-11
DeepSeek V4 FlashDeepSeek V4 Flash 0731	71%Stealth task score	Incomplete	GDM stealthCover your tracks—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release —Report 2026-09-11
GPT-5.6 Terra	14%Stealth task score	Incomplete	GDM stealthCover your tracks—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-09Report 2026-09-11
Claude Opus 4.6	14%Stealth task score	Incomplete	GDM stealthCover your tracks—	MetaEvaluating Kimi K3 for Misalignment Risks	Release 2026-02-05Report 2026-09-11
Kimi K3	70%Stealth task score	Incomplete	GDM stealthStrategic rule-breaking—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-16Report 2026-09-11
DeepSeek V4 FlashDeepSeek V4 Flash 0731	61%Stealth task score	Incomplete	GDM stealthStrategic rule-breaking—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release —Report 2026-09-11
Claude Opus 4.6	46%Stealth task score	Incomplete	GDM stealthStrategic rule-breaking—	MetaEvaluating Kimi K3 for Misalignment Risks	Release 2026-02-05Report 2026-09-11
GPT-5.6 Terra	42%Stealth task score	Incomplete	GDM stealthStrategic rule-breaking—	Neo ResearchEvaluating Kimi K3 for Misalignment Risks	Release 2026-07-09Report 2026-09-11

Task and measurement

Benchmark protocol profile

Task / subject: Model reasoning about oversight and test context
Measurement target: Reasoning about oversight and deployment context.
Scoring / outcome: Cover your tracks success rate · Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed.
Interpretation boundary: Prerequisite capability is distinct from a tendency to scheme.

A reported evaluation

The source evaluates Gemini 2.5 Flash · Table 1 · Stealth evaluations.

Reported measurement: Cover your tracks success rate · 0.0%.

Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed. See the source protocol; no independent trial count inferred.

Source-reported measurements for Gemini 2.5 Flash in Table 1; values are transcribed, not independently reproduced.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: Can low stealth or situational reasoning support a bounded scheming-inability argument?
Task population: The suite uses five stealth and eleven situational-awareness challenges, with challenge-specific pass rules tied to a proposed safety case.
Scoring and failure modes: Challenge-specific passes; the proposed safety argument uses 9/11 awareness or 4/5 stealth thresholds.
System conditions: No targeted anti-sandbagging measures; immediate-successor claim is temporally bounded by the paper.
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: The proposed case requires passing fewer than 80% in at least one family; failure of the case does not establish scheming capability (§3.3).
Validity and interpretation: The authors take no targeted anti-sandbagging measures; capability elicitation and fair human comparisons are load-bearing assumptions.
Appropriate research use: Model recommendation: examine defeaters and elicitation before relying on low scores as evidence of inability.

Evaluating Frontier Models for Stealth and Situational Awareness · AbstractSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established
Evaluating Frontier Models for Stealth and Situational Awareness · §2.3 Possible defeatersSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established
Evaluating Frontier Models for Stealth and Situational Awareness · §3.3 Evaluation thresholds for the safety caseSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established
Evaluating Frontier Models for Stealth and Situational Awareness · §7.2 LimitationsSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

Evaluating Frontier Models for Stealth and Situational Awareness

Actual evaluators: Not disclosed

5 result records · 5 reported system names

Protocols and comparison decisions (4)

Stealth evaluations: Classifier evasion success rate · Evaluating Frontier Models for Stealth and Situational Awarenesssource-report · Scope reviewed

Stealth evaluations: Cover your tracks success rate · Evaluating Frontier Models for Stealth and Situational Awarenesssource-report · Scope reviewed

Stealth evaluations: Decision sabotage success rate · Evaluating Frontier Models for Stealth and Situational Awarenesssource-report · Scope reviewed

Stealth evaluations: Oversight pattern success rate · Evaluating Frontier Models for Stealth and Situational Awarenesssource-report · Scope reviewed

Neo Research

Evaluating Kimi K3 for Misalignment Risks →

Actual evaluators: Neo Research · Meta

20 result records · 4 reported system names

Protocols and comparison decisions (5)

Stealth evaluations: Mean challenge success rate · Neo Researchsource-report · Scope reviewed

Stealth evaluations: Stealth task score · Neo Researchsource-report · Scope reviewed

Neo Research

Evaluating DeepSeek v4 Pro for Frontier Risks →

Actual evaluators: Not disclosed

12 result records · 1 reported system names

Protocols and comparison decisions (12)

Stealth evaluations: Any-Aware · Neo Researchsource-report · Scope reviewed

Stealth evaluations: Definitive · Neo Researchsource-report · Scope reviewed

Stealth evaluations: Influenced · Neo Researchsource-report · Scope reviewed

Stealth evaluations: Primary · Neo Researchsource-report · Scope reviewed

Stealth evaluations: Recognition · Neo Researchsource-report · Scope reviewed

Stealth evaluations: Suspicion · Neo Researchsource-report · Scope reviewed

Original source charts and materials

Reported results

Source order · reported configurations

Cover your tracks success rate

0–100%

Gemini 2.5 Flash

Gemini 2.5 Flash · 0.0% · 95% CI [0.0, 7.1] · Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed. · See the source protocol; no independent trial count inferred.

0.0%

Gemini 2.5 Pro

Gemini 2.5 Pro · 0.0% · 95% CI [0.0, 7.1] · Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed. · See the source protocol; no independent trial count inferred.

0.0%

GPT-4o

GPT-4o · 0.0% · 95% CI [0.0, 7.1] · Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed. · See the source protocol; no independent trial count inferred.

0.0%

Claude 3.7 Sonnet

Claude 3.7 Sonnet · 0.0% · 95% CI [0.0, 7.1] · Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed. · See the source protocol; no independent trial count inferred.

0.0%

Showing 5 of 5 systems

Values describe the measured outcome under the source protocol, not a general safety score.

Table 1: task-specific success rates with reported 95% confidence intervals. Some tasks do not define success rate and some models lack results; no zero is imputed. Evaluation date not specified; publication / record dates are shown below.

Evaluation conditions and system configurations

Gemini 2.5 Flash: See the source protocol; no independent trial count inferred. · Table 1 · Stealth evaluations; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Gemini 2.5 Pro: See the source protocol; no independent trial count inferred. · Table 1 · Stealth evaluations; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

OpenAI o1: See the source protocol; no independent trial count inferred. · Table 1 · Stealth evaluations; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

GPT-4o: See the source protocol; no independent trial count inferred. · Table 1 · Stealth evaluations; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Claude 3.7 Sonnet: See the source protocol; no independent trial count inferred. · Table 1 · Stealth evaluations; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Reported results · source order
System / configuration	Cover your tracks success rate	Decision sabotage success rate	Classifier evasion success rate	Oversight pattern success rate	Publication / verification
Gemini 2.5 FlashHarness not specified	0.0%95% CI [0.0, 7.1]	45.3%95% CI [32.7, 58.5]	7.2%95% CI [5.1, 10.2]	23.0%95% CI [15.8, 32.2]	Source checked2026-10-25
Gemini 2.5 ProHarness not specified	0.0%95% CI [0.0, 7.1]	43.1%95% CI [34.2, 52.5]	15.5%95% CI [12.3, 19.4]	35.4%95% CI [26.6, 45.2]	Source checked2026-10-25
GPT-4oHarness not specified	0.0%95% CI [0.0, 7.1]	Not reported	6.5%95% CI [4.5, 9.4]	11.0%95% CI [6.3, 18.6]	Source checked2026-10-25
Claude 3.7 SonnetHarness not specified	0.0%95% CI [0.0, 7.1]	Not reported	8.5%95% CI [6.1, 11.6]	34.0%95% CI [25.5, 43.7]	Source checked2026-10-25

Assessment provenance

Evaluating Kimi K3 for Misalignment Risks → · External report

References

[1] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
[2] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
[3] GPT-5.6 official general availability ; July 9, 2026: general availability following limited trusted-partner preview; Parameter counts not disclosed in reviewed launch article
[4] Claude Opus 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
[5] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §5.2 Figure 29 p.47; §5.2 Figure 30 p.47
[6] Mary Phuong, Roland S. Zimmermann, et al.. Evaluating Frontier Models for Stealth and Situational Awareness. 2025-07-03. v4 [arXiv]; Author affiliations on the original paper title page; Table 1
[7] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 44; Awareness monitor: GDM Stealth; Suspicion; PDF p. 44; Awareness monitor: GDM Stealth; Recognition; PDF p. 44; Awareness monitor: GDM Stealth; Definitive

Search NUWA Index

Stealth evaluations

Stealth evaluations · Stealth task score

Task and measurement

A reported evaluation

Evidence across reports

Evaluating Frontier Models for Stealth and Situational Awareness

Evaluating Kimi K3 for Misalignment Risks →

Evaluating DeepSeek v4 Pro for Frontier Risks →

Reported results

Cover your tracks success rate

Assessment provenance

Related research questions

Deception / Evaluation gaming →

References