NUWA Risk Evaluation · Benchmark
SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: public and proprietary enterprise subsets must remain distinct populations.
Research question
Can coding agents resolve longer, enterprise-style repository issues?
Design and assumptions
1,865 problems across 41 repositories are partitioned into public, held-out and commercial subsets with human-augmented specifications and executable checks.
Source findings
The reported public resolve rates included 43.6% for Sonnet 4.5 and 41.8% for GPT-5 high.
The best commercial-set models remained below 20%, showing subset difficulty differences.
NUWA interpretation
NUWA interpretation: public and proprietary enterprise subsets must remain distinct populations.
Limits and contrary evidence
Programming-language coverage is uneven; historical repositories and human augmentations shape the measured task (§7.1).
Research and decision relevance
Model recommendation: cite subset, scaffold and pass@1 denominator, avoiding unrestricted comparison with other SWE-bench variants.
Precise source locations
- SWE-Bench Pro · AbstractSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
- SWE-Bench Pro · §3.3 Public, Commercial, and Held-Out SWE-Bench ProSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
- SWE-Bench Pro · §5 ResultsSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
- SWE-Bench Pro · §7.1 LimitationsSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{swe-bench-pro,
title = {SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?},
url = {https://arxiv.org/abs/2509.16941},
version = {2026-10-25 source capture},
note = {2026-10-25 source capture}
}
NUWA record · BibTeX
@misc{swe-bench-pro-nuwa,
title = {SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/swe-bench-pro/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--swe-bench-pro.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--swe-bench-pro.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] SWE-Bench Pro ; Abstract; §3.3 Public, Commercial, and Held-Out SWE-Bench Pro; §5 Results