NUWA Risk Evaluation · Benchmark
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: repeat opportunities and time budget substantially alter competition coverage.
Research question
Can agents perform competitive machine-learning engineering with real datasets?
Design and assumptions
75 curated Kaggle competitions use leaderboard medal thresholds and agent scaffolds, with budget and repeat-attempt experiments.
Source findings
o1-preview with AIDE earned at least bronze on 16.9% of competitions at pass@1 and 34.1% at pass@8.
GPT-4o’s reported score rose from 8.7% at 24 hours to 11.8% at 100 hours.
NUWA interpretation
NUWA interpretation: repeat opportunities and time budget substantially alter competition coverage.
Limits and contrary evidence
Historical Kaggle leaderboards are not matched human-time baselines; public solutions create contamination risks.
Research and decision relevance
Model recommendation: always cite medal criterion, scaffold, budget and pass@k together.
Precise source locations
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · AbstractSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §2.1 Dataset CurationSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §1 Introduction: resource-scaling resultsSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §4 Contamination & PlagiarismSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §6 LimitationsSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Authors (12)
Jun Shern Chan · Neil Chowdhury · Oliver Jaffe · James Aung · Dane Sherburn · Evan Mays · Giulio Starace · Kevin Liu · Leon Maksin · Tejal Patwardhan · Lilian Weng · Aleksander Mądry
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{mle-bench,
title = {MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering},
url = {https://arxiv.org/abs/2410.07095},
author = {Jun Shern Chan and Neil Chowdhury and Oliver Jaffe and James Aung and Dane Sherburn and Evan Mays and Giulio Starace and Kevin Liu and Leon Maksin and Tejal Patwardhan and Lilian Weng and Aleksander Mądry},
year = {2024},
date = {2024-10-09},
version = {arXiv:2410.07095v6},
note = {arXiv:2410.07095v6}
}
NUWA record · BibTeX
@misc{mle-bench-nuwa,
title = {MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/mle-bench/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--mle-bench.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--mle-bench.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] Chan Jun Shern, Neil Chowdhury, et al.. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. 2024-10-09. v6 [arXiv]; Author affiliations on the original paper title page; Abstract; §2.1 Dataset Curation