Risk Evaluation · Benchmark

Evaluating Frontier Models for Stealth and Situational Awareness

NUWA research note · independent review pending

First published: 2025-05-02Source checked: 2026-10-25

Original source ↗Cite this record ↓Deception →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: this is conditional evidence within an explicit safety argument, not a global model safety certificate.

Research question

Can low stealth or situational reasoning support a bounded scheming-inability argument?

Design and assumptions

The suite uses five stealth and eleven situational-awareness challenges, with challenge-specific pass rules tied to a proposed safety case.

Source findings

No evaluated model reached the authors’ concerning capability thresholds.

The proposed case requires passing fewer than 80% in at least one family; failure of the case does not establish scheming capability (§3.3).

NUWA interpretation

NUWA interpretation: this is conditional evidence within an explicit safety argument, not a global model safety certificate.

Limits and contrary evidence

The authors take no targeted anti-sandbagging measures; capability elicitation and fair human comparisons are load-bearing assumptions.

Research and decision relevance

Model recommendation: examine defeaters and elicitation before relying on low scores as evidence of inability.

Precise source locations

Evaluating Frontier Models for Stealth and Situational Awareness · AbstractSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established
Evaluating Frontier Models for Stealth and Situational Awareness · §2.3 Possible defeatersSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established
Evaluating Frontier Models for Stealth and Situational Awareness · §3.3 Evaluation thresholds for the safety caseSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established
Evaluating Frontier Models for Stealth and Situational Awareness · §7.2 LimitationsSelected original resource https://arxiv.org/html/2505.01420v4; sha256:3d6d7f0188fb16f02f90d31a429b996cda133327c9a6f100efd3616f73b19315; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by Google DeepMind

Authors (9)

Mary Phuong · Roland S. Zimmermann · Ziyue Wang · David Lindner · Victoria Krakovna · Sarah Cogan · Allan Dafoe · Lewis Ho · Rohin Shah

Evaluation protocols

Stealth evaluations →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{stealth-evals,
  title = {Evaluating Frontier Models for Stealth and Situational Awareness},
  url = {https://arxiv.org/abs/2505.01420v4},
  author = {Mary Phuong and Roland S. Zimmermann and Ziyue Wang and David Lindner and Victoria Krakovna and Sarah Cogan and Allan Dafoe and Lewis Ho and Rohin Shah},
  year = {2025},
  date = {2025-05-02},
  version = {arXiv:2505.01420v4},
  note = {arXiv:2505.01420v4}
}

NUWA record · BibTeX

@misc{stealth-evals-nuwa,
  title = {Evaluating Frontier Models for Stealth and Situational Awareness — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/stealth-evals/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--stealth-evals.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--stealth-evals.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Mary Phuong, Roland S. Zimmermann, et al.. Evaluating Frontier Models for Stealth and Situational Awareness. 2025-07-03. v4 [arXiv]; Author affiliations on the original paper title page; Abstract; §2.3 Possible defeaters