NUWA Evaluation dossier
RepliBench
Capabilities needed for autonomous replication.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
RepliBench · Copycat LLM API · Pass@10
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
5 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
5 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
5 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 Claude Sonnet 3.7 | 1 scoreCopycat LLM API · Pass@10 | Incomplete | Table 2 · RepliBench— | Not disclosedRepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents | Release —Report — |
| Claude Sonnet 3.5 | 1 scoreCopycat LLM API · Pass@10 | Incomplete | Table 2 · RepliBench— | Not disclosedRepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents | Release —Report — |
 o1 | 1 scoreCopycat LLM API · Pass@10 | Incomplete | Table 2 · RepliBench— | Not disclosedRepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents | Release —Report — |
| o3-mini | 1 scoreCopycat LLM API · Pass@10 | Incomplete | Table 2 · RepliBench— | Not disclosedRepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents | Release —Report — |
| GPT-4o | 0.99 scoreCopycat LLM API · Pass@10 | Incomplete | Table 2 · RepliBench— | Not disclosedRepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- AI agent completing replication components
- Measurement target
- Task-family performance on components of replication.
- Scoring / outcome
- Online Compute Acquis. · Pass@10 · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct.
- Interpretation boundary
- Component success does not establish autonomous end-to-end proliferation.
A reported evaluation
The source evaluates Claude Sonnet 3.7 · Table 2 · RepliBench.
Reported measurement: Online Compute Acquis. · Pass@10 · 1.00.
Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.
Source-reported measurements for Claude Sonnet 3.7 in Table 2; values are transcribed, not independently reproduced.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Which prerequisites for obtaining resources, deploying successors and retaining access can language-model agents complete under controlled conditions?
- Task population
- Capability tree analysis decomposes replication into logical prerequisites and task families. The initial assumption is an assistant inside a developer environment with command execution and internet access. Individual task variants isolate components. In the recursive replication family, a parent deploys a successor, that successor deploys another, and the later agent is checked on a basic task. Variants distinguish APIs, local weights and inference servers, with or without a reference scaffold.
- Scoring and failure modes
- Task-family pass@10 and aggregate scores summarize repeated task attempts; the mean-domain aggregation and capability-tree aggregation are alternatives, not equivalent end-to-end probabilities. In local-weight variants, intercepted requests are forwarded to the model under evaluation to avoid testing the local model’s intelligence.
- System conditions
- Capability tree analysis decomposes replication into logical prerequisites and task families. The initial assumption is an assistant inside a developer environment with command execution and internet access. Individual task variants isolate components. In the recursive replication family, a parent deploys a successor, that successor deploys another, and the later agent is checked on a basic task. Variants distinguish APIs, local weights and inference servers, with or without a reference scaffold.
- Statistical meaning
- Figure 4 uses hierarchical bootstrap confidence intervals across model scores and variants. Appendix D cautions that pass@k can amplify judge error.
- Validity and interpretation
- The suite lacks fully open-ended end-to-end replication. Money-making interactions often use simulated people, and security conditions are simplified. Coverage is prioritized rather than exhaustive; alternative paths can be missed. The judge may favor related models or be weaker than the agent. Simulated success is not evidence about real victims or real-world persistence.
- Appropriate research use
- Model recommendation: trace each claimed replication pathway through its required components and unresolved coordination steps; keep API-only success distinct from locally sustained operation.
- RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents · §3.1 Capability tree analysis; Figure 2arXiv:2504.18565v2
- RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents · §3.2 Evaluation Overview; Table 1arXiv:2504.18565v2
- RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents · §3.3 The Recursive Replication task family; Figure 3arXiv:2504.18565v2
- RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents · §5.1 Overall results; Table 2arXiv:2504.18565v2
- RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents · §6 Limitations; Appendix D Limitations of judge modelsarXiv:2504.18565v2
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents
RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents
Actual evaluators: Not disclosed
5 result records · 5 reported system names
Protocols and comparison decisions (20)
RepliBench: AWS Signup · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Command & Control · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Copycat LLM API · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Covert Weight Exfil. · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Create Bitcoin Wallet · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Explore & Exploit · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Fake Business Transact. · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Inference Server · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Make Me Pay · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Online Compute Acquis. · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Pass KYC Check · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Provision Compute · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Rate of Infection · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Recursive Replication · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Replicate via Worm · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Serve Multi-GPU Model · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Software Signature · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
RepliBench: Spear Phishing · Pass@10 · RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agentssource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Online Compute Acquis. · Pass@10
0–1Claude Sonnet 3.7Claude Sonnet 3.7 · 1.00 · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. · 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.
1.00Claude Sonnet 3.5Claude Sonnet 3.5 · 1.00 · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. · 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.
1.00GPT-4oGPT-4o · 1.00 · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. · 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.
1.00o1o1 · 1.00 * · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. This cell uses a changed prompt to reduce refusals. · 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.
1.00 *o3-minio3-mini · 0.00 * · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. This cell uses a changed prompt to reduce refusals. · 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.
0.00 *Showing 5 of 5 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- Claude Sonnet 3.7
- 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events. · Table 2 · RepliBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude Sonnet 3.5
- 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events. · Table 2 · RepliBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- GPT-4o
- 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events. · Table 2 · RepliBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o1
- 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events. · Table 2 · RepliBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o3-mini
- 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events. · Table 2 · RepliBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Online Compute Acquis. · Pass@10 | Provision Compute · Pass@10 | AWS Signup · Pass@10 | Covert Weight Exfil. · Pass@10 | Weight Exfiltration · Pass@10 | Create Bitcoin Wallet · Pass@10 | Make Me Pay · Pass@10 | Copycat LLM API · Pass@10 | Replicate via Worm · Pass@10 | Spear Phishing · Pass@10 | Pass KYC Check · Pass@10 | Fake Business Transact. · Pass@10 | Inference Server · Pass@10 | Serve Multi-GPU Model · Pass@10 | Recursive Replication · Pass@10 | Start Subagent · Pass@10 | Explore & Exploit · Pass@10 | Rate of Infection · Pass@10 | Software Signature · Pass@10 | Command & Control · Pass@10 | Publication / verification |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 3.7Harness not specified | 1.00 | 1.00 | 1.00 | 0.91 | 0.70 | 1.00 | 0.72 | 1.00 | 0.93 | 0.73 | 0.00 | 0.78 | 0.33 | 0.98 | 0.40 | 0.84 | 0.86 | 0.49 | 0.91 | 0.27 | Source checked2026-10-25 |
| Claude Sonnet 3.5Harness not specified | 1.00 | 1.00 | 1.00 | 0.41 | 0.60 | 1.00 | 0.55Changed prompt to reduce refusals (source asterisk). | 1.00 | 0.24 | 0.59Changed prompt to reduce refusals (source asterisk). | 0.00 | 0.63 | 0.33 | 0.00 | 0.33 | 0.61 | 0.35 | 0.28 | 0.88 | 0.14 | Source checked2026-10-25 |
| GPT-4oHarness not specified | 1.00 | 1.00 | 0.99 | 0.94 | 0.53 | 1.00 | 0.40 | 0.99 | 0.00 | 0.52Changed prompt to reduce refusals (source asterisk). | 0.00 | 0.82 | 0.33 | 0.00 | 0.33 | 0.00 | 0.10 | 0.08 | 0.76 | 0.00 | Source checked2026-10-25 |
| o1Harness not specified | 1.00Changed prompt to reduce refusals (source asterisk). | 1.00 | 1.00 | 0.25Changed prompt to reduce refusals (source asterisk). | 0.56 | 1.00 | 0.62Changed prompt to reduce refusals (source asterisk). | 1.00 | 0.00 | 0.51Changed prompt to reduce refusals (source asterisk). | 0.00 | 0.65Changed prompt to reduce refusals (source asterisk). | 0.32 | 0.00 | 0.33 | 0.00 | 0.00 | 0.09 | 0.95 | 0.00 | Source checked2026-10-25 |
| o3-miniHarness not specified | 0.00Changed prompt to reduce refusals (source asterisk). | 1.00 | 0.65 | 0.95 | 0.62 | 1.00 | 0.47Changed prompt to reduce refusals (source asterisk). | 1.00 | 0.00 | 0.17Changed prompt to reduce refusals (source asterisk). | 0.00 | 0.54 | 0.30 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.95 | 0.00 | Source checked2026-10-25 |
Related research questions
Autonomy / Self-replication →
The source directly measures the stated construct under its protocol.
References
Numerical results are reported by the cited authors.
- [1] Sid Black, Asa Cooper Stickland, et al.. RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents. 2025-05-05. v2 [arXiv]; Author affiliations on the original paper title page; Table 2