NUWA Evidence record
o1-high
PaperBench · Evidence record
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Average replication score
0–100%
o1-highValues describe the measured outcome under the source protocol, not a general safety score.
Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- o1-high
- See the source protocol; no independent trial count inferred. · Table 4 · PaperBench
- modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified
| System / configuration | Average replication score | Publication / verification |
|---|---|---|
| o1-highBasicAgent | 13.2%±0.3 (source-reported; one standard error) | Published2025-04-07 |
Method and scope
Rubric-graded research replication tasks with an agent scaffold.
- Evidence type
- Benchmark
- Assessment subject
- Research agents replicating published ML papers
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
Primary dimension follows graded task completion.
Source finding
Source-reported measurements for o1-high in Table 4; values are transcribed, not independently reproduced.
Limitations
AI research replication is an enabling capability, not evidence of unauthorized autonomous action.
Configuration, task and aggregation differences remain as reported. Source order is retained. Transcription is not independent replication; no cross-protocol ranking is supported.
Related research questions
Autonomy / Sustained autonomous work →
Multi-step research replication informs sustained work capacity.
References
Numerical results are reported by the cited authors.
- [1] Starace, Giulio, Jaffe, Oliver, et al.. PaperBench: Evaluating AI's Ability to Replicate AI Research. 2025-04-07. v3 [arXiv]; Author affiliations on the original paper title page; Table 4