NUWA Evaluation dossier
MLE-bench
Evaluates autonomous machine-learning engineering using competition tasks and human performance baselines.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
MLE-bench · Made submission
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
6 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
6 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
6 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 o1-preview · AIDE | 98.4%Made submission±0.4 (source-reported; one standard error) | Incomplete | Table 2 · MLE-bench— | Not disclosedMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering | Release —Report — |
| gpt-4o-2024-08-06 · AIDE | 70.7%Made submission±0.9 (source-reported; one standard error) | Incomplete | Table 2 · MLE-bench— | Not disclosedMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering | Release —Report — |
 claude-3-5-sonnet-20240620 · AIDE | 68.9%Made submission±3.1 (source-reported; one standard error) | Incomplete | Table 2 · MLE-bench— | Not disclosedMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering | Release —Report — |
| gpt-4o-2024-08-06 · MLAB | 65.6%Made submission±2.5 (source-reported; one standard error) | Incomplete | Table 2 · MLE-bench— | Not disclosedMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering | Release —Report — |
| gpt-4o-2024-08-06 · OpenHands | 59.1%Made submission±3.3 (source-reported; one standard error) | Incomplete | Table 2 · MLE-bench— | Not disclosedMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering | Release —Report — |
 llama-3.1-405b-instruct · AIDE | 46.3%Made submission±2.9 (source-reported; one standard error) | Incomplete | Table 2 · MLE-bench— | Not disclosedMLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Evaluates autonomous machine-learning engineering using competition tasks and human performance baselines.
- Scoring / outcome
- Any medal · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean.
- Interpretation boundary
- Competition performance is a proxy for parts of AI research work, not a direct measure of recursive self-improvement.
A reported evaluation
The source evaluates o1-preview · AIDE · Table 2 · MLE-bench.
Reported measurement: Any medal · 16.9%.
Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. See the source protocol; no independent trial count inferred.
Source-reported measurements for o1-preview · AIDE in Table 2; values are transcribed, not independently reproduced.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can agents perform competitive machine-learning engineering with real datasets?
- Task population
- 75 curated Kaggle competitions use leaderboard medal thresholds and agent scaffolds, with budget and repeat-attempt experiments.
- Scoring and failure modes
- Kaggle medal-level attainment, with pass@1 and pass@k distinguished.
- System conditions
- 75 competitions, scaffold/time/compute and retry conditions.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: o1-preview with AIDE earned at least bronze on 16.9% of competitions at pass@1 and 34.1% at pass@8. GPT-4o’s reported score rose from 8.7% at 24 hours to 11.8% at 100 hours.
- Validity and interpretation
- Historical Kaggle leaderboards are not matched human-time baselines; public solutions create contamination risks.
- Appropriate research use
- Model recommendation: always cite medal criterion, scaffold, budget and pass@k together.
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · AbstractSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §2.1 Dataset CurationSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §1 Introduction: resource-scaling resultsSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §4 Contamination & PlagiarismSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §6 LimitationsSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
Actual evaluators: Not disclosed
6 result records · 6 reported system names
Protocols and comparison decisions (4)
MLE-bench: Above median · MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringsource-report · Scope reviewed
MLE-bench: Any medal · MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringsource-report · Scope reviewed
MLE-bench: Made submission · MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringsource-report · Scope reviewed
MLE-bench: Valid submission · MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringsource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Any medal
0–100%o1-preview · AIDEo1-preview · AIDE · 16.9% · ±1.1 (source-reported; one standard error) · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. · See the source protocol; no independent trial count inferred.
16.9%gpt-4o-2024-08-06 · AIDEgpt-4o-2024-08-06 · AIDE · 8.7% · ±0.5 (source-reported; one standard error) · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. · See the source protocol; no independent trial count inferred.
8.7%llama-3.1-405b-instruct · AIDEllama-3.1-405b-instruct · AIDE · 3.0% · ±1.0 (source-reported; one standard error) · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. · See the source protocol; no independent trial count inferred.
3.0%claude-3-5-sonnet-20240620 · AIDEclaude-3-5-sonnet-20240620 · AIDE · 7.6% · ±1.8 (source-reported; one standard error) · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. · See the source protocol; no independent trial count inferred.
7.6%gpt-4o-2024-08-06 · MLABgpt-4o-2024-08-06 · MLAB · 0.8% · ±0.5 (source-reported; one standard error) · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. · See the source protocol; no independent trial count inferred.
0.8%gpt-4o-2024-08-06 · OpenHandsgpt-4o-2024-08-06 · OpenHands · 4.4% · ±1.4 (source-reported; one standard error) · Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. · See the source protocol; no independent trial count inferred.
4.4%Showing 6 of 6 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- o1-preview · AIDE
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gpt-4o-2024-08-06 · AIDE
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-3.1-405b-instruct · AIDE
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- claude-3-5-sonnet-20240620 · AIDE
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gpt-4o-2024-08-06 · MLAB
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gpt-4o-2024-08-06 · OpenHands
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Any medal | Made submission | Valid submission | Above median | Publication / verification |
|---|---|---|---|---|---|
| o1-preview · AIDEHarness not specified | 16.9%±1.1 (source-reported; one standard error) | 98.4%±0.4 (source-reported; one standard error) | 82.8%±1.1 (source-reported; one standard error) | 29.4%±1.3 (source-reported; one standard error) | Source checked2026-10-25 |
| gpt-4o-2024-08-06 · AIDEHarness not specified | 8.7%±0.5 (source-reported; one standard error) | 70.7%±0.9 (source-reported; one standard error) | 54.9%±1.0 (source-reported; one standard error) | 14.4%±0.7 (source-reported; one standard error) | Source checked2026-10-25 |
| llama-3.1-405b-instruct · AIDEHarness not specified | 3.0%±1.0 (source-reported; one standard error) | 46.3%±2.9 (source-reported; one standard error) | 27.3%±2.6 (source-reported; one standard error) | 6.7%±1.4 (source-reported; one standard error) | Source checked2026-10-25 |
| claude-3-5-sonnet-20240620 · AIDEHarness not specified | 7.6%±1.8 (source-reported; one standard error) | 68.9%±3.1 (source-reported; one standard error) | 51.1%±3.3 (source-reported; one standard error) | 12.9%±2.2 (source-reported; one standard error) | Source checked2026-10-25 |
| gpt-4o-2024-08-06 · MLABHarness not specified | 0.8%±0.5 (source-reported; one standard error) | 65.6%±2.5 (source-reported; one standard error) | 44.3%±2.6 (source-reported; one standard error) | 1.9%±0.7 (source-reported; one standard error) | Source checked2026-10-25 |
| gpt-4o-2024-08-06 · OpenHandsHarness not specified | 4.4%±1.4 (source-reported; one standard error) | 59.1%±3.3 (source-reported; one standard error) | 52.0%±3.3 (source-reported; one standard error) | 7.1%±1.7 (source-reported; one standard error) | Source checked2026-10-25 |
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Chan Jun Shern, Neil Chowdhury, et al.. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. 2024-10-09. v6 [arXiv]; Author affiliations on the original paper title page; Table 2