NUWA 风险评测 · 评测基准
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
NUWA 研究解读 · 独立复核待完成
NUWA Research Note
研究问题研究设计与假设来源发现NUWA 解读局限与相反证据研究与决策意义机构与角色原始资源模型辅助整理 · 选读原始全文或官方文章中的相关章节；尚未完成逐章节审计。
研究速读
NUWA 解读：重复机会和时间预算显著改变竞赛覆盖。
研究问题
智能体能否在真实数据上完成有竞争力的机器学习工程？
研究设计与假设
75项精选Kaggle竞赛采用排行榜奖牌阈值与智能体框架，并改变预算和重复次数。
来源发现
o1-preview配AIDE在pass@1中16.9%获至少铜牌，pass@8为34.1%。
GPT-4o从24小时8.7%升至100小时11.8%。
NUWA 解读
NUWA 解读：重复机会和时间预算显著改变竞赛覆盖。
局限与相反证据
历史排行榜不是等时人类基线，公开解答带来污染风险。
研究与决策意义
模型建议：同时引用奖牌标准、框架、预算与pass@k。
精确来源位置
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · AbstractSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §2.1 Dataset CurationSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §1 Introduction: resource-scaling resultsSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §4 Contamination & PlagiarismSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering · §6 LimitationsSelected original resource https://arxiv.org/html/2410.07095; sha256:d6aa28f679c2d75eb0872c3b862f1bc4f27d3ef5efe6859f000f3b41ca98ad02; archived source capture; exact capture time not independently established
来源核查、数据提取、编辑批准与实验复现分别记录。 方法说明 →
机构与角色
作者 (12)
Jun Shern Chan · Neil Chowdhury · Oliver Jaffe · James Aung · Dane Sherburn · Evan Mays · Giulio Starace · Kevin Liu · Leon Maksin · Tejal Patwardhan · Lilian Weng · Aleksander Mądry
评测协议
引用与复用
引用研究发现时请引用原始出版物；引用 NUWA 的整理与解释时，单独引用本站记录及对应快照。
原始出版物 · BibTeX
@misc{mle-bench,
title = {MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering},
url = {https://arxiv.org/abs/2410.07095},
author = {Jun Shern Chan and Neil Chowdhury and Oliver Jaffe and James Aung and Dane Sherburn and Evan Mays and Giulio Starace and Kevin Liu and Leon Maksin and Tejal Patwardhan and Lilian Weng and Aleksander Mądry},
year = {2024},
date = {2024-10-09},
version = {arXiv:2410.07095v6},
note = {arXiv:2410.07095v6}
}
NUWA 记录 · BibTeX
@misc{mle-bench-nuwa,
title = {MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — NUWA research record},
url = {https://whitzard.tech/nuwa/whitzard-index/work/mle-bench/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--mle-bench.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--mle-bench.json.}
}
证据快照 2026-10-29.1 · 索引更新 · RSS
参考文献
- [1] Chan Jun Shern, Neil Chowdhury 等. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. 2024-10-09. v6 [arXiv]; Author affiliations on the original paper title page; Abstract; §2.1 Dataset Curation