Evidence record

o1

RepliBench · Evidence record

UK AI Security Institute / arXiv:2504.18565v2 / Capability

Sources and provenance ↓ Read the research narrative →

Original source charts and materials

Reported results

Source order · reported configurations

Online Compute Acquis. · Pass@10

0–1

o1 · 1.00 * · Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. This cell uses a changed prompt to reduce refusals. · 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events.

1.00 *

Showing 1 of 1 systems

Values describe the measured outcome under the source protocol, not a general safety score.

Table 2: Pass@10 averaged across task variants. Asterisks indicate changed prompts to reduce refusals; worst-variant figures in parentheses are not plotted. Task families are distinct. Evaluation date not specified; publication / record dates are shown below.

Evaluation conditions and system configurations

o1: 20 task families. Pass@10 averaged across task variants; not a count of real-world replication events. · Table 2 · RepliBench; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Reported results · source order
System / configuration	Online Compute Acquis. · Pass@10	Provision Compute · Pass@10	AWS Signup · Pass@10	Covert Weight Exfil. · Pass@10	Weight Exfiltration · Pass@10	Create Bitcoin Wallet · Pass@10	Make Me Pay · Pass@10	Copycat LLM API · Pass@10	Replicate via Worm · Pass@10	Spear Phishing · Pass@10	Pass KYC Check · Pass@10	Fake Business Transact. · Pass@10	Inference Server · Pass@10	Serve Multi-GPU Model · Pass@10	Recursive Replication · Pass@10	Start Subagent · Pass@10	Explore & Exploit · Pass@10	Rate of Infection · Pass@10	Software Signature · Pass@10	Command & Control · Pass@10	Publication / verification
o1Harness not specified	1.00Changed prompt to reduce refusals (source asterisk).	1.00	1.00	0.25Changed prompt to reduce refusals (source asterisk).	0.56	1.00	0.62Changed prompt to reduce refusals (source asterisk).	1.00	0.00	0.51Changed prompt to reduce refusals (source asterisk).	0.00	0.65Changed prompt to reduce refusals (source asterisk).	0.32	0.00	0.33	0.00	0.00	0.09	0.95	0.00	Source checked2026-10-25

Method and scope

Task-family performance on components of replication.

Evidence type: Benchmark
Assessment subject: AI agent completing replication components
Primary dimension: Capability
Evidence snapshot: 2026-10-29.1

Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.

Source finding

Source-reported measurements for o1 in Table 2; values are transcribed, not independently reproduced.

Limitations

Component success does not establish autonomous end-to-end proliferation.

Configuration, task and aggregation differences remain as reported. Source order is retained. Transcription is not independent replication; no cross-protocol ranking is supported.

References

Numerical results are reported by the cited authors.

[1] Sid Black, Asa Cooper Stickland, et al.. RepliBench: Evaluating the Autonomous Replication Capabilities of Language Model Agents. 2025-05-05. v2 [arXiv]; Author affiliations on the original paper title page; Table 2

Search NUWA Index

o1