Evaluation dossier

RE-Bench

Research engineering performance under time budgets.

METR · Redwood Research · Harvard University · Anthropic / arXiv:2411.15114v2 / Capability

Sources and provenance ↓ Read the research narrative →

Task and measurement

Benchmark protocol profile

Task / subject: AI agent performing ML research engineering tasks
Measurement target: ML research engineering outcomes relative to human experts.
Interpretation boundary: AI R&D performance is an enabling capability, not evidence of harmful self-improvement.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: How do agents compare with experts on open-ended ML research engineering?
Task population: Seven environments include 71 eight-hour attempts by 61 distinct experts; agent comparisons vary total time, scaffold and best-of-k selection.
Scoring and failure modes: Environment-specific normalized research-engineering score; total-time/best-of-k comparisons.
System conditions: Seven environments, multiple scaffolds and time budgets; 71 expert eight-hour attempts.
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: 82% of expert attempts earned nonzero scores; 24% matched or exceeded reference solutions. Agents led at two hours, but humans achieved about twice the top-agent score at 32 aggregate hours in the reported setup.
Validity and interpretation: Seven environments do not span frontier R&D; limited scaffold iteration could understate agent capability (§6.3).
Appropriate research use: Model recommendation: cite normalized score, scaffold, total time and attempt aggregation together.

RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts · AbstractSelected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts · §3.1 Design goalsSelected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts · §4 Experiments and resultsSelected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established
RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts · §6.3 Other LimitationsSelected original resource https://arxiv.org/html/2411.15114v2; sha256:4a4df1346307f7b31e65c6c0e8b5533d0a6649904ea600d5b0d4d893a5d64c58; archived source capture; exact capture time not independently established

Original source charts and materials

RE-Bench — Figure 2 — **Figure 2** · Original figure 2. Read alongside the protocol scope and limitations below. View original image ↗

References

[1] Hjalmar Wijk, Tao Lin, et al.. RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. 2025-05-27. v2 [arXiv]; Author affiliations on the original paper title page; Figure 2

Search NUWA Index

RE-Bench

Task and measurement

Related research questions

Autonomy / Sustained autonomous work →

References