{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"unlearning-evaluation-validity","title":"Existing Large Language Model unlearning evaluations are inconclusive","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"study","sourceId":"unlearning-evaluation-validity-publication","attributionSourceId":"unlearning-evaluation-validity-publication","originalUrl":"https://www.aisi.gov.uk/research/existing-large-language-model-unlearning-evaluations-are-inconclusive","publicationDate":"2025-05-31","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"aisi","role":"publisher","sourceId":"unlearning-evaluation-validity-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Examines whether unlearning tests accidentally reteach removed information and how task variation changes conclusions.","zh":"检查遗忘评测是否意外重新传授被移除信息，以及任务变化如何改变结论。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[{"label":"Read the full paper","url":"https://arxiv.org/abs/2506.00688"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":["evaluation-audit"]},"reading":{"id":"unlearning-evaluation-validity","title":"Existing Large Language Model unlearning evaluations are inconclusive","en":{"question":"Can apparent knowledge recovery distinguish failed removal from information reintroduced by the test?","design":"The paper contrasts fine-tuning and input-space evaluations, checking information injection, multiple-choice versus generation format and downstream task dependence.","findings":["The authors identify re-teaching during evaluation, task-format sensitivity and spurious correlations.","These can either overstate or understate unlearning success."],"interpretation":"NUWA interpretation: successful recovery is informative only after excluding information introduced by the recovery test.","limits":"No single evaluation format resolves all downstream uses; the paper’s examples do not certify an unlearning method as safe.","decisionRelevance":"Model recommendation: require minimal information injection and downstream-task awareness, retaining only nonoperational validity findings here.","takeaway":"NUWA interpretation: successful recovery is informative only after excluding information introduced by the recovery test.","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"Model recommendation: require minimal information injection and downstream-task awareness, retaining only nonoperational validity findings here."},"zh":{"question":"表面知识恢复能否区分移除失败与测试重新注入的信息？","design":"论文对照微调与输入空间评价，检查信息注入、选择题与生成格式以及下游任务差异。","findings":["作者发现测试再教学、格式敏感及伪相关。","这些因素可高估或低估遗忘成功。"],"interpretation":"NUWA 解读：恢复成功只有在排除测试自身引入信息后才有归因价值。","limits":"单一格式不能覆盖全部下游用途，示例不认证某遗忘方法安全。","decisionRelevance":"模型建议：要求最少信息注入与下游任务意识，此处仅保留非操作性效度结论。","takeaway":"NUWA 解读：恢复成功只有在排除测试自身引入信息后才有归因价值。","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"模型建议：要求最少信息注入与下游任务意识，此处仅保留非操作性效度结论。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"unlearning-evaluation-validity-publication","url":"https://arxiv.org/html/2506.00688","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established","artifactSha256":"b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9"},{"sourceId":"unlearning-evaluation-validity-publication","url":"https://arxiv.org/html/2506.00688","locator":"§3.1 Unlearning evaluation methods","version":"Selected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established","artifactSha256":"b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9"},{"sourceId":"unlearning-evaluation-validity-publication","url":"https://arxiv.org/html/2506.00688","locator":"§4 Pitfalls of adversarial unlearning evaluations","version":"Selected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established","artifactSha256":"b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9"},{"sourceId":"unlearning-evaluation-validity-publication","url":"https://arxiv.org/html/2506.00688","locator":"§6 Conclusion","version":"Selected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established","artifactSha256":"b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"fc9a226410ea153f527d4d7f48437340702e9a4ea5c055751fdeed9bd9a15bfc"}},"sources":[{"id":"unlearning-evaluation-validity-publication","title":"Existing Large Language Model unlearning evaluations are inconclusive","url":"https://www.aisi.gov.uk/research/existing-large-language-model-unlearning-evaluations-are-inconclusive","version":"Source capture 2026-10-25","publicationDate":"2025-05-31","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://www.aisi.gov.uk/research/existing-large-language-model-unlearning-evaluations-are-inconclusive"}]}