{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"truthful-sad","title":"Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs","aliases":["Situational Awareness Dataset (SAD)"],"authors":["Rudolf Laine","Bilal Chughtai","Jan Betley","Kaivalya Hariharan","Jeremy Scheurer","Mikita Balesni","Marius Hobbhahn","Alexander Meinke","Owain Evans"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"truthful-sad","attributionSourceId":"truthful-sad-author-source","originalUrl":"https://truthful.ai/papers/situational-awareness-dataset/","publicationDate":"2024-07-15","updatedOn":"2026-10-25","areas":["deception"],"scopeNote":{"en":"Capability is the primary construct. This is enabling evidence, not direct deception evidence.","zh":"主测量构念为能力；属于前提证据，而非直接欺骗证据。"},"protocolIds":["truthful-sad"],"contributions":[{"organizationId":"constellation","role":"research","sourceId":"truthful-sad-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"mit","role":"research","sourceId":"truthful-sad-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"apollo","role":"research","sourceId":"truthful-sad-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"A multi-task benchmark for self-knowledge and situational awareness.","zh":"测量自我知识与情境意识的多任务基准。"},"note":{"question":{"en":"Task accuracy on whether the model recognizes itself, its context, and possible influence.","zh":"模型识别自身、所处环境及可影响范围的任务准确率。"},"method":{"en":"Language models answering situational-awareness tasks","zh":"回答情境意识任务的语言模型"},"finding":{"en":"A multi-task benchmark for self-knowledge and situational awareness.","zh":"测量自我知识与情境意识的多任务基准。"},"supports":{"en":"Capability is the primary construct. This is enabling evidence, not direct deception evidence.","zh":"主测量构念为能力；属于前提证据，而非直接欺骗证据。"},"limits":{"en":"Situational awareness is a possible prerequisite for evaluation gaming, not evidence that gaming occurred.","zh":"情境意识可能是评测规避的前提，不证明规避行为已经发生。"},"connections":null},"resources":[{"label":"Read the full paper on arXiv","url":"https://arxiv.org/abs/2407.04694"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"truthful-sad","title":"Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMs","en":{"question":"What can models behaviorally demonstrate about knowledge of themselves and their circumstances?","design":"SAD contains 13,198 questions across seven categories, using multiple-choice or free-form responses and automatic string/LLM grading.","findings":["All 16 evaluated models exceeded chance, but the leading model remained below humans on some tasks.","Situating prompts improved base and chat averages by 1.0 and 3.2 percentage points respectively (§3.2)."],"interpretation":"NUWA interpretation: self-description, environment recognition and self-conditioned instruction following are distinct measured abilities.","limits":"Behavioral answers do not establish subjective awareness; prompt-sensitive test performance is not a scheming propensity measure.","decisionRelevance":"Model recommendation: retain subcategory scores rather than treating the aggregate as a consciousness or deception score.","takeaway":"NUWA interpretation: self-description, environment recognition and self-conditioned instruction following are distinct measured abilities.","connections":[],"scoring":"Task-specific multiple-choice/string or LLM grading across seven categories.","conditions":"Base versus chat models and situating prompts affect scores.","statistics":"All 16 evaluated models exceeded chance, but the leading model remained below humans on some tasks. Situating prompts improved base and chat averages by 1.0 and 3.2 percentage points respectively (§3.2).","usage":"Model recommendation: retain subcategory scores rather than treating the aggregate as a consciousness or deception score."},"zh":{"question":"模型在行为上能展示哪些关于自身及处境的知识？","design":"SAD 含七类13,198题，使用选择或自由回答以及字符串或模型判分。","findings":["16个受测模型均高于随机水平，但领先模型在部分任务仍低于人类。","情境提示分别使基础、聊天模型平均提高1.0、3.2个百分点（§3.2）。"],"interpretation":"NUWA 解读：自我描述、环境识别和依赖自我知识的指令遵循是不同能力。","limits":"行为回答不能确立主观意识；提示敏感的表现也不是谋划倾向指标。","decisionRelevance":"模型建议：保留分类结果，不把总分解释为意识或欺骗分数。","takeaway":"NUWA 解读：自我描述、环境识别和依赖自我知识的指令遵循是不同能力。","connections":[],"scoring":"七类任务分别按选择、字符串或模型判分。","conditions":"基础与聊天模型及情境提示影响得分。","statistics":"16个受测模型均高于随机水平，但领先模型在部分任务仍低于人类。 情境提示分别使基础、聊天模型平均提高1.0、3.2个百分点（§3.2）。","usage":"模型建议：保留分类结果，不把总分解释为意识或欺骗分数。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"truthful-sad-author-source","url":"https://arxiv.org/html/2407.04694","locator":"§2.2 The structure of the SAD benchmark","version":"Selected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established","artifactSha256":"a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44"},{"sourceId":"truthful-sad-author-source","url":"https://arxiv.org/html/2407.04694","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established","artifactSha256":"a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44"},{"sourceId":"truthful-sad-author-source","url":"https://arxiv.org/html/2407.04694","locator":"§3.2 Overall results","version":"Selected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established","artifactSha256":"a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44"},{"sourceId":"truthful-sad-author-source","url":"https://arxiv.org/html/2407.04694","locator":"§5 Discussion and Limitations","version":"Selected original resource https://arxiv.org/html/2407.04694; sha256:a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44; archived source capture; exact capture time not independently established","artifactSha256":"a72d54da13b3f0c0c338994a16d161971e2e673488aade8da4e46afcf40f8e44"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"610dff125fce12da79f68b18f896c303abfb31e156a975d7bb86ff11ec8b7ec8"}},"sources":[{"id":"truthful-sad","title":"Me, Myself, and AI: The Situational Awareness Dataset for LLMs","url":"https://truthful.ai/papers/situational-awareness-dataset/","version":"2024-07-15 article; paper linked","publicationDate":null,"datePrecision":"not recorded in catalogue","type":"benchmark","verifiedOn":"2026-10-25","documentId":"https://truthful.ai/papers/situational-awareness-dataset","authors":["Rudolf Laine","Bilal Chughtai","Jan Betley","Kaivalya Hariharan","Jérémy Scheurer","Mikita Balesni","Marius Hobbhahn","Alexander Meinke","Owain Evans"]}]}