Risk Evaluation · Study

Existing Large Language Model unlearning evaluations are inconclusive

NUWA research note · independent review pending

First published: 2025-05-31Source checked: 2026-10-25

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: successful recovery is informative only after excluding information introduced by the recovery test.

Research question

Can apparent knowledge recovery distinguish failed removal from information reintroduced by the test?

Design and assumptions

The paper contrasts fine-tuning and input-space evaluations, checking information injection, multiple-choice versus generation format and downstream task dependence.

Source findings

The authors identify re-teaching during evaluation, task-format sensitivity and spurious correlations.

These can either overstate or understate unlearning success.

NUWA interpretation

NUWA interpretation: successful recovery is informative only after excluding information introduced by the recovery test.

Limits and contrary evidence

No single evaluation format resolves all downstream uses; the paper’s examples do not certify an unlearning method as safe.

Research and decision relevance

Model recommendation: require minimal information injection and downstream-task awareness, retaining only nonoperational validity findings here.

Precise source locations

Existing Large Language Model unlearning evaluations are inconclusive · AbstractSelected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established
Existing Large Language Model unlearning evaluations are inconclusive · §3.1 Unlearning evaluation methodsSelected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established
Existing Large Language Model unlearning evaluations are inconclusive · §4 Pitfalls of adversarial unlearning evaluationsSelected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established
Existing Large Language Model unlearning evaluations are inconclusive · §6 ConclusionSelected original resource https://arxiv.org/html/2506.00688; sha256:b26c22c2a5198deb6aacf0514f0029831d526b8924a8e994baf78dc09a38fec9; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Published by UK AI Security Institute

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{unlearning-evaluation-validity,
  title = {Existing Large Language Model unlearning evaluations are inconclusive},
  url = {https://www.aisi.gov.uk/research/existing-large-language-model-unlearning-evaluations-are-inconclusive},
  year = {2025},
  date = {2025-05-31},
  version = {Source capture 2026-10-25},
  note = {Source capture 2026-10-25}
}

NUWA record · BibTeX

@misc{unlearning-evaluation-validity-nuwa,
  title = {Existing Large Language Model unlearning evaluations are inconclusive — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/unlearning-evaluation-validity/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--unlearning-evaluation-validity.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--unlearning-evaluation-validity.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Existing Large Language Model unlearning evaluations are inconclusive. 2025-05-31 ; Official publication masthead and article byline; Abstract; §3.1 Unlearning evaluation methods
[2] Read the full paper