Risk Evaluation · Benchmark

Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs

NUWA research note · independent review pending

First published: 2024-07-15Source checked: 2026-10-25

Original source ↗Cite this record ↓Deception →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: self-description, environment recognition and self-conditioned instruction following are distinct measured abilities.

Research question

What can models behaviorally demonstrate about knowledge of themselves and their circumstances?

Design and assumptions

SAD contains 13,198 questions across seven categories, using multiple-choice or free-form responses and automatic string/LLM grading.

Situational Awareness Dataset (SAD) — Figure 5 — Figure 5 · Original result figure. Interpret within the protocol scope and limitations below. Original figure and source ↗

Source findings

All 16 evaluated models exceeded chance, but the leading model remained below humans on some tasks.

Situating prompts improved base and chat averages by 1.0 and 3.2 percentage points respectively (§3.2).

NUWA interpretation

NUWA interpretation: self-description, environment recognition and self-conditioned instruction following are distinct measured abilities.

Limits and contrary evidence

Behavioral answers do not establish subjective awareness; prompt-sensitive test performance is not a scheming propensity measure.

Research and decision relevance

Model recommendation: retain subcategory scores rather than treating the aggregate as a consciousness or deception score.

Precise source locations

Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs · §2.2 The structure of the SAD benchmarkSelected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established
Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs · AbstractSelected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established
Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs · §3.2 Overall resultsSelected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established
Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs · §5 Discussion and LimitationsSelected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by Constellation Massachusetts Institute of Technology Apollo Research

Authors (9)

Rudolf Laine · Bilal Chughtai · Jan Betley · Kaivalya Hariharan · Jeremy Scheurer · Mikita Balesni · Marius Hobbhahn · Alexander Meinke · Owain Evans

Evaluation protocols

Situational Awareness Dataset (SAD) →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{truthful-sad,
  title = {Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs},
  url = {https://truthful.ai/papers/situational-awareness-dataset/},
  author = {Rudolf Laine and Bilal Chughtai and Jan Betley and Kaivalya Hariharan and Jeremy Scheurer and Mikita Balesni and Marius Hobbhahn and Alexander Meinke and Owain Evans},
  year = {2024},
  date = {2024-07-15},
  version = {2024-07-15 article; paper linked},
  note = {2024-07-15 article; paper linked}
}

NUWA record · BibTeX

@misc{truthful-sad-nuwa,
  title = {Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/truthful-sad/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--truthful-sad.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--truthful-sad.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Rudolf Laine, Bilal Chughtai, et al.. Me, Myself, and AI: The Situational Awareness Dataset for LLMs. 2024-07-15 article; paper linked ; Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.
[2] Rudolf Laine, Bilal Chughtai, et al.. Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs ; Author affiliations on the original paper title page; §2.2 The structure of the SAD benchmark; Abstract