NUWA Evidence record
o1-preview · AIDE
MLE-bench · Evidence record
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Any medal
0–100%
o1-preview · AIDEValues describe the measured outcome under the source protocol, not a general safety score.
Table 2: scaffolding and model experiments, generally 3 seeds; AIDE o1-preview uses 16 and AIDE GPT-4o uses 36. ± is one standard error of the mean. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- o1-preview · AIDE
- See the source protocol; no independent trial count inferred. · Table 2 · MLE-bench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Any medal | Made submission | Valid submission | Above median | Publication / verification |
|---|---|---|---|---|---|
| o1-preview · AIDEHarness not specified | 16.9%±1.1 (source-reported; one standard error) | 98.4%±0.4 (source-reported; one standard error) | 82.8%±1.1 (source-reported; one standard error) | 29.4%±1.3 (source-reported; one standard error) | Source checked2026-10-25 |
Method and scope
Evaluates autonomous machine-learning engineering using competition tasks and human performance baselines.
- Evidence type
- Benchmark
- Assessment subject
- Model or agent system under the original task configuration
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Source finding
Source-reported measurements for o1-preview · AIDE in Table 2; values are transcribed, not independently reproduced.
Limitations
Competition performance is a proxy for parts of AI research work, not a direct measure of recursive self-improvement.
Configuration, task and aggregation differences remain as reported. Source order is retained. Transcription is not independent replication; no cross-protocol ranking is supported.
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Chan Jun Shern, Neil Chowdhury, et al.. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. 2024-10-09. v6 [arXiv]; Author affiliations on the original paper title page; Table 2