NUWA Risk Governance · Study
Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch connectionsResearch and decision relevanceEvidence updateInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
Pretraining curation improved resistance on knowledge proxies, with important limits when external information is available.
Research question
Can withholding domain knowledge at pretraining provide a more durable safeguard than post-training alone?
Design and assumptions
The authors train matched-architecture models on filtered/unfiltered corpora, compare general and proxy-knowledge evaluations, and test bounded tampering resistance; this review excludes hazardous task content.
Source findings
Filtered models preserve measured unrelated capabilities while reducing the targeted proxy knowledge.
The paper explicitly finds that contextual information changes the safeguard boundary; no tested defense is presented as universally robust.
NUWA interpretation
NUWA interpretation: resistance on internalized-knowledge proxies supports one defense layer, not proof that tool-augmented misuse is prevented.
Limits and contrary evidence
Model scale, proxy construct, training distribution and tested adversary resources bound the result. Real-world harmful outcomes are not directly measured.
Research connections
NUWA connection: open-weight-risk-management identifies curation and tampering evaluation as separate research gaps; RAND’s perspective questions different links between proxies and outcomes.
Research and decision relevance
NUWA recommendation: report proxy validity and external-information assumptions with any safeguard claim; require independent safety review before broader deployment inference.
Precise source locations
- Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs · 2 Filtering Prevents Target Capabilities: evaluation designSelected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established
- Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs · 3 Filtering Achieves State-of-the-Art Tamper-Resistance: aggregate resultsSelected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established
- Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs · 4 Defense-in-Depth: reported limitationsSelected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established
- Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs · 6.3 LimitationsSelected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established
- Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs · 6.4 Impact and Risk StatementSelected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
Check initial effect, selectivity and durability under later adaptation separately. Protein-model filtering and tool reconstruction address different assumptions in a safety argument; neither is a same-model replication of Deep Ignorance, and proxy scores are not observed biological harm.
Related reading
NUWA reading connections; these do not imply citation, replication or score equivalence.
Sources for this interpretation
- ReproBAIT: Can coding agents build their own biological AI tools? · Abstract; Table2 caption; §3; limitations; text lines 00207Original resource captured 2026-10-28T07:25:52.920583+00:00; sha256:cdc5d63c54b48afc059393a7b154a5670ae73b72361536695ada5890520234f1
- The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models · Abstract; §4.4 limitations; conclusion; text lines 00271Original resource captured 2026-10-28T07:25:52.921030+00:00; sha256:83a3c96f121b7de5bcb6ba87f388d6ac39e33416fd00b70cbcf68d3235ac7670
Institutions and roles
Authors (10)
Kyle O'Brien · Stephen Casper · Quentin Anthony · Tomek Korbak · Robert Kirk · Xander Davies · Ishan Mishra · Geoffrey Irving · Yarin Gal · Stella Biderman
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{deep-ignorance,
title = {Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs},
url = {https://arxiv.org/abs/2508.06601},
author = {Kyle O'Brien and Stephen Casper and Quentin Anthony and Tomek Korbak and Robert Kirk and Xander Davies and Ishan Mishra and Geoffrey Irving and Yarin Gal and Stella Biderman},
year = {2025},
date = {2025-08-08},
version = {arXiv:2508.06601v2},
note = {arXiv:2508.06601v2}
}
NUWA record · BibTeX
@misc{deep-ignorance-nuwa,
title = {Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/deep-ignorance/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--deep-ignorance.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--deep-ignorance.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] Kyle O’Brien, Stephen Casper, et al.. Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs. 2025-08-08. v2 [arXiv]; Author affiliations on the original paper title page; 2 Filtering Prevents Target Capabilities: evaluation design; 3 Filtering Achieves State-of-the-Art Tamper-Resistance: aggregate results