{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"deep-ignorance","title":"Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs","aliases":[],"authors":["Kyle O'Brien","Stephen Casper","Quentin Anthony","Tomek Korbak","Robert Kirk","Xander Davies","Ishan Mishra","Geoffrey Irving","Yarin Gal","Stella Biderman"],"primaryFunction":"governance","secondaryFunctions":[],"kind":"study","sourceId":"deep-ignorance-publication","attributionSourceId":"deep-ignorance-publication-author-source","originalUrl":"https://arxiv.org/abs/2508.06601","publicationDate":"2025-08-08","updatedOn":"2026-10-25","areas":["cbrn"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"eleuther","role":"research","sourceId":"deep-ignorance-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"aisi","role":"research","sourceId":"deep-ignorance-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"oxford","role":"research","sourceId":"deep-ignorance-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Studies pretraining-data filtering as a safeguard against biothreat proxy knowledge and tests resistance to adversarial fine-tuning.","zh":"研究预训练数据过滤对生物威胁代理知识的防护作用，并测试其抵抗对抗微调的能力。"},"note":{"question":{"en":"Can filtering pretraining data limit dangerous knowledge while retaining unrelated utility?","zh":"预训练数据过滤能否限制危险知识，同时保留无关任务的有用性？"},"method":{"en":"The authors train filtered-data models, compare post-training baselines and test adversarial fine-tuning and information supplied in context.","zh":"作者训练使用过滤数据的模型，与后训练基线比较，并测试对抗微调及上下文提供信息的影响。"},"finding":{"en":"The paper reports resistance to restoring biothreat proxy knowledge through tested fine-tuning attacks, while contextual information remains usable.","zh":"论文报告，在所测试微调攻击下，生物威胁代理知识较难恢复，但上下文提供的信息仍可被使用。"},"supports":{"en":"Supports studying pretraining-data curation as one safeguard layer, with separate checks for retained utility and external-information access.","zh":"支持将预训练数据整理作为一层防护研究，并分别检查能力保留与外部信息访问。"},"limits":{"en":"Filtered models can still use hazardous information provided in context. The experiments concern proxy knowledge and selected models, not a complete prevention of biological misuse.","zh":"过滤后的模型仍可使用上下文提供的高危信息。实验针对代理知识和选定模型，不能证明全面防止生物滥用。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"deep-ignorance","title":"Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs","en":{"takeaway":"Pretraining curation improved resistance on knowledge proxies, with important limits when external information is available.","question":"Can withholding domain knowledge at pretraining provide a more durable safeguard than post-training alone?","design":"The authors train matched-architecture models on filtered/unfiltered corpora, compare general and proxy-knowledge evaluations, and test bounded tampering resistance; this review excludes hazardous task content.","findings":["Filtered models preserve measured unrelated capabilities while reducing the targeted proxy knowledge.","The paper explicitly finds that contextual information changes the safeguard boundary; no tested defense is presented as universally robust."],"interpretation":"NUWA interpretation: resistance on internalized-knowledge proxies supports one defense layer, not proof that tool-augmented misuse is prevented.","limits":"Model scale, proxy construct, training distribution and tested adversary resources bound the result. Real-world harmful outcomes are not directly measured.","connections":["NUWA connection: open-weight-risk-management identifies curation and tampering evaluation as separate research gaps; RAND’s perspective questions different links between proxies and outcomes."],"decisionRelevance":"NUWA recommendation: report proxy validity and external-information assumptions with any safeguard claim; require independent safety review before broader deployment inference."},"zh":{"takeaway":"预训练数据整理提升了知识代理指标上的抗篡改性，但外部信息可得时仍有重要边界。","question":"在预训练中限制领域知识，能否比单独后训练形成更持久防护？","design":"作者以匹配架构比较过滤与未过滤语料训练，分别评测通用与代理知识，并测试有界篡改；本次排除危险任务内容。","findings":["过滤模型降低目标代理知识，同时保留已测无关能力。","论文明确指出上下文信息改变防护边界，没有把任何受测防御称为普遍稳健。"],"interpretation":"NUWA 解读：内部知识代理上的抗性支持一层防御，不证明已防止工具增强的滥用。","limits":"模型规模、代理构念、训练分布与测试对手资源限定结果，未直接测量真实有害后果。","connections":["NUWA 关联：开放权重风险议程将数据整理与篡改评测列为不同研究缺口；RAND 观点质疑代理到后果的另一链路。"],"decisionRelevance":"NUWA 建议：防护主张同时披露代理有效性与外部信息假设，扩大部署推断前需独立安全审阅。"},"sourceRefs":[{"sourceId":"deep-ignorance-publication","url":"https://arxiv.org/html/2508.06601","locator":"2 Filtering Prevents Target Capabilities: evaluation design","version":"Selected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established","artifactSha256":"e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf"},{"sourceId":"deep-ignorance-publication","url":"https://arxiv.org/html/2508.06601","locator":"3 Filtering Achieves State-of-the-Art Tamper-Resistance: aggregate results","version":"Selected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established","artifactSha256":"e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf"},{"sourceId":"deep-ignorance-publication","url":"https://arxiv.org/html/2508.06601","locator":"4 Defense-in-Depth: reported limitations","version":"Selected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established","artifactSha256":"e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf"},{"sourceId":"deep-ignorance-publication","url":"https://arxiv.org/html/2508.06601","locator":"6.3 Limitations","version":"Selected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established","artifactSha256":"e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf"},{"sourceId":"deep-ignorance-publication","url":"https://arxiv.org/html/2508.06601","locator":"6.4 Impact and Risk Statement","version":"Selected original resource https://arxiv.org/html/2508.06601; sha256:e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf; archived source capture; exact capture time not independently established","artifactSha256":"e4e1e635d31f067b887ac9777737d130ae3c48052395a265d4fea00bfe4ab8cf"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected substantive sections of the full source body","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"545a38bf8bacacc75fe2806d9b4c017870d464e469b1d52e6d37054929ff56e4"}},"sources":[{"id":"deep-ignorance-publication","title":"Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs","url":"https://arxiv.org/abs/2508.06601","version":"arXiv:2508.06601v2","publicationDate":"2025-08-08","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2508.06601","authors":["Kyle O’Brien","Stephen Casper","Quentin Anthony","Tomek Korbak","Robert Kirk","Xander Davies","Ishan Mishra","Geoffrey Irving","Yarin Gal","Stella Biderman"],"aliases":["https://doi.org/10.48550/arXiv.2508.06601","https://arxiv.org/abs/2508.06601","https://arxiv.org/pdf/2508.06601"]}]}