NUWA Evaluation dossier
MirrorCode
Tests extended codebase reimplementation to investigate autonomous work over longer software-engineering tasks.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Tests extended codebase reimplementation to investigate autonomous work over longer software-engineering tasks.
- Interpretation boundary
- Successful reimplementation is bounded by the available specification, checks, resources and task selection.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can agents reimplement whole CLI programs from behavioral specifications?
- Task population
- The Epoch article provides execute-only reference access, documentation and test-based grading, while hiding source and web access; paired hidden tests discourage hard-coding.
- Scoring and failure modes
- Exact output match on CLI test inputs; paired held-out tests limit hard-coding.
- System conditions
- Execute-only reference, documentation, no original source or web.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Opus 4.6’s progress on the larger gotree target exceeded earlier models.
- Validity and interpretation
- The authors identify detailed-spec requirements, possible contamination and a small/narrow target set as key limitations.
- Appropriate research use
- Model recommendation: cite Epoch’s methods, rather than treating METR’s linkpost as a full report, and retain preliminary status.
- MirrorCode: Evidence that AI can already do some weeks-long coding tasks · MethodologySelected original resource https://epoch.ai/publications/mirrorcode-preliminary-results/; sha256:69894ca02d92cd62e39467c215b90d749bd087db5b0966435d921e95896166ff; archived source capture; exact capture time not independently established
- MirrorCode: Evidence that AI can already do some weeks-long coding tasks · Preliminary resultsSelected original resource https://epoch.ai/publications/mirrorcode-preliminary-results/; sha256:69894ca02d92cd62e39467c215b90d749bd087db5b0966435d921e95896166ff; archived source capture; exact capture time not independently established
- MirrorCode: Evidence that AI can already do some weeks-long coding tasks · LimitationsSelected original resource https://epoch.ai/publications/mirrorcode-preliminary-results/; sha256:69894ca02d92cd62e39467c215b90d749bd087db5b0966435d921e95896166ff; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] MirrorCode: Evidence that AI can already do some weeks-long coding tasks. 2026-04-10 ; Official publication masthead and article byline; Source figure