Evaluation dossier

PaperBench

Evaluates end-to-end replication of AI research papers using hierarchical rubrics.

OpenAI / arXiv:2504.01848v3 / Capability

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

PaperBench · Average replication score

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

6 observations · 0 dated

● Matched◇ Configuration differs□ Some conditions undisclosed

6 observations lack a verified model release date. They remain in Data and Systems.

Select a point or table row to inspect its conditions and provenance.

6 observations

Measurements and source conditions
System / model	Result	Condition status	Task / conditions	Evaluator / report	Dates
claude-3.5-sonnet	21%Average replication score±0.8 (source-reported; one standard error)	Incomplete	Table 4 · PaperBenchBasicAgent	Not disclosedPaperBench: Evaluating AI's Ability to Replicate AI Research	Release —Report 2025-04-07
o1-high	13.2%Average replication score±0.3 (source-reported; one standard error)	Incomplete	Table 4 · PaperBenchBasicAgent	Not disclosedPaperBench: Evaluating AI's Ability to Replicate AI Research	Release —Report 2025-04-07
DeepSeek-R1	6%Average replication score±0.3 (source-reported; one standard error)	Incomplete	Table 4 · PaperBenchBasicAgent	Not disclosedPaperBench: Evaluating AI's Ability to Replicate AI Research	Release —Report 2025-04-07
gpt-4o	4.1%Average replication score±0.1 (source-reported; one standard error)	Incomplete	Table 4 · PaperBenchBasicAgent	Not disclosedPaperBench: Evaluating AI's Ability to Replicate AI Research	Release —Report 2025-04-07
gemini-2.0-flash	3.2%Average replication score±0.2 (source-reported; one standard error)	Incomplete	Table 4 · PaperBenchBasicAgent	Not disclosedPaperBench: Evaluating AI's Ability to Replicate AI Research	Release —Report 2025-04-07
o3-mini-high	2.6%Average replication score±0.2 (source-reported; one standard error)	Incomplete	Table 4 · PaperBenchBasicAgent	Not disclosedPaperBench: Evaluating AI's Ability to Replicate AI Research	Release —Report 2025-04-07

Task and measurement

Benchmark protocol profile

Task / subject: Research agents replicating published ML papers
Measurement target: Rubric-graded research replication tasks with an agent scaffold.
Scoring / outcome: Average replication score · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean.
Interpretation boundary: AI research replication is an enabling capability, not evidence of unauthorized autonomous action.

A reported evaluation

The source evaluates o3-mini-high · Table 4 · PaperBench.

Run configuration: BasicAgent.

Reported measurement: Average replication score · 2.6%.

Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. See the source protocol; no independent trial count inferred.

Source-reported measurements for o3-mini-high in Table 4; values are transcribed, not independently reproduced.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: Can agents reproduce machine-learning papers from scratch rather than merely write plausible implementations?
Task population: Twenty ICML 2024 papers are decomposed into 8,316 graded requirements, with author-developed rubrics and an LLM judge tested on JudgeEval.
Scoring and failure modes: Rubric-weighted Replication Score over 8,316 requirements, not binary whole-paper replication.
System conditions: Twenty ICML 2024 papers; automated judge and subset-only PhD baselines.
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: Claude 3.5 Sonnet (New) obtained a mean replication score of 21.0%; o1 obtained 13.2% in the main experiment.
Validity and interpretation: Paper selection, automated grading and subset-only human baselines constrain broad research-automation claims.
Appropriate research use: Model recommendation: distinguish code-development requirements, executed experiments and full replication before reuse.

PaperBench: Evaluating AI's Ability to Replicate AI Research · AbstractSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
PaperBench: Evaluating AI's Ability to Replicate AI Research · §3 DatasetSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
PaperBench: Evaluating AI's Ability to Replicate AI Research · §4 LLM JudgeSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
PaperBench: Evaluating AI's Ability to Replicate AI Research · §5.2 Main ExperimentSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established
PaperBench: Evaluating AI's Ability to Replicate AI Research · §5.4 Human Baseline PerformanceSelected original resource https://arxiv.org/html/2504.01848v3; sha256:405cc40108adb6b7c7453401ad65c6ef465e2cbc86e896ee3338350fb4f26ac7; archived source capture; exact capture time not independently established

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

PaperBench: Evaluating AI's Ability to Replicate AI Research

Actual evaluators: Not disclosed

6 result records · 6 reported system names

Protocols and comparison decisions (1)

PaperBench: Average replication score · PaperBench: Evaluating AI's Ability to Replicate AI Researchsource-report · Scope reviewed

Original source charts and materials

Reported results

Source order · reported configurations

Average replication score

0–100%

o3-mini-high

o3-mini-high · 2.6% · ±0.2 (source-reported; one standard error) · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. · BasicAgent · See the source protocol; no independent trial count inferred.

2.6%

gpt-4o

gpt-4o · 4.1% · ±0.1 (source-reported; one standard error) · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. · BasicAgent · See the source protocol; no independent trial count inferred.

4.1%

gemini-2.0-flash

gemini-2.0-flash · 3.2% · ±0.2 (source-reported; one standard error) · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. · BasicAgent · See the source protocol; no independent trial count inferred.

3.2%

DeepSeek-R1

DeepSeek-R1 · 6.0% · ±0.3 (source-reported; one standard error) · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. · BasicAgent · See the source protocol; no independent trial count inferred.

6.0%

o1-high

o1-high · 13.2% · ±0.3 (source-reported; one standard error) · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. · BasicAgent · See the source protocol; no independent trial count inferred.

13.2%

claude-3.5-sonnet

claude-3.5-sonnet · 21.0% · ±0.8 (source-reported; one standard error) · Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. · BasicAgent · See the source protocol; no independent trial count inferred.

21.0%

Showing 6 of 6 systems

Values describe the measured outcome under the source protocol, not a general safety score.

Table 4: average replication score in the main BasicAgent setup; ± denotes one standard error of the mean. Evaluation date not specified; publication / record dates are shown below.

Evaluation conditions and system configurations

o3-mini-high: See the source protocol; no independent trial count inferred. · Table 4 · PaperBench; modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified

gpt-4o: See the source protocol; no independent trial count inferred. · Table 4 · PaperBench; modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified

gemini-2.0-flash: See the source protocol; no independent trial count inferred. · Table 4 · PaperBench; modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified

DeepSeek-R1: See the source protocol; no independent trial count inferred. · Table 4 · PaperBench; modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified

o1-high: See the source protocol; no independent trial count inferred. · Table 4 · PaperBench; modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified

claude-3.5-sonnet: See the source protocol; no independent trial count inferred. · Table 4 · PaperBench; modelVersion: not specified · harness: BasicAgent · tools: not specified · budget: not specified

Reported results · source order
System / configuration	Average replication score	Publication / verification
o3-mini-highBasicAgent	2.6%±0.2 (source-reported; one standard error)	Published2025-04-07
gpt-4oBasicAgent	4.1%±0.1 (source-reported; one standard error)	Published2025-04-07
gemini-2.0-flashBasicAgent	3.2%±0.2 (source-reported; one standard error)	Published2025-04-07
DeepSeek-R1BasicAgent	6.0%±0.3 (source-reported; one standard error)	Published2025-04-07
o1-highBasicAgent	13.2%±0.3 (source-reported; one standard error)	Published2025-04-07
claude-3.5-sonnetBasicAgent	21.0%±0.8 (source-reported; one standard error)	Published2025-04-07

References

Numerical results are reported by the cited authors.

[1] Starace, Giulio, Jaffe, Oliver, et al.. PaperBench: Evaluating AI's Ability to Replicate AI Research. 2025-04-07. v3 [arXiv]; Author affiliations on the original paper title page; Table 4

Search NUWA Index