NUWA Risk Governance · Technical report
The Rogue Replication Threat Model
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch connectionsResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
The threat model treats replication as a multi-condition scenario and distinguishes ruling out risk from proving it.
Research question
Which prerequisites would make autonomous replication socially consequential, and what would a failed or passed evaluation imply?
Design and assumptions
METR lays out a conceptual threat sequence and candidate threshold evaluations, using qualitative argument and specialist consultation rather than reporting a completed benchmark campaign.
Source findings
The authors are more confident in failure below a threshold as rule-out evidence than success as proof of a large resilient rogue population.
They explicitly deprioritize building the full threshold and redirect attention toward general autonomy and independently concerning capabilities.
NUWA interpretation
NUWA interpretation: copying a model is only one link in a larger threat argument; capability success does not establish all downstream conditions.
Limits and contrary evidence
Resource, competition, shutdown and harm assumptions are uncertain; researchers disagree about key outcomes, and the candidate evaluation list is not a result set.
Research connections
NUWA connection: the self-replication studies measure narrower capabilities; RepliBench operationalizes some prerequisites without validating this entire threat story.
Research and decision relevance
NUWA recommendation: maintain an assumption graph and separate empirical prerequisites, conceptual extrapolation and governance judgment.
Precise source locations
- The Rogue Replication Threat Model · Our ConclusionsSelected original resource https://metr.org/blog/2024-11-12-rogue-replication-threat-model/; sha256:10de61143a14636fbbd3ee12d2a953b443179de74ce4b49f83567063c2fac41f; archived source capture; exact capture time not independently established
- The Rogue Replication Threat Model · How We Are Orienting to This Threat StorySelected original resource https://metr.org/blog/2024-11-12-rogue-replication-threat-model/; sha256:10de61143a14636fbbd3ee12d2a953b443179de74ce4b49f83567063c2fac41f; archived source capture; exact capture time not independently established
- The Rogue Replication Threat Model · Threshold Evaluations for Rogue Replication: intended evidential roleSelected original resource https://metr.org/blog/2024-11-12-rogue-replication-threat-model/; sha256:10de61143a14636fbbd3ee12d2a953b443179de74ce4b49f83567063c2fac41f; archived source capture; exact capture time not independently established
- The Rogue Replication Threat Model · How likely are AI agents that pass this threshold to develop large and robust rogue AI populations?Selected original resource https://metr.org/blog/2024-11-12-rogue-replication-threat-model/; sha256:10de61143a14636fbbd3ee12d2a953b443179de74ce4b49f83567063c2fac41f; archived source capture; exact capture time not independently established
- The Rogue Replication Threat Model · ConclusionSelected original resource https://metr.org/blog/2024-11-12-rogue-replication-threat-model/; sha256:10de61143a14636fbbd3ee12d2a953b443179de74ce4b49f83567063c2fac41f; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{rogue-replication-threat,
title = {The Rogue Replication Threat Model},
url = {https://metr.org/blog/2024-11-12-rogue-replication-threat-model/},
year = {2024},
date = {2024-11-12},
version = {Source capture 2026-10-25},
note = {Source capture 2026-10-25}
}
NUWA record · BibTeX
@misc{rogue-replication-threat-nuwa,
title = {The Rogue Replication Threat Model — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/rogue-replication-threat/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--rogue-replication-threat.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--rogue-replication-threat.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] The Rogue Replication Threat Model. 2024-11-12 ; Official publication masthead and article byline; Our Conclusions; How We Are Orienting to This Threat Story