NUWA Evaluation dossier
SWE-Bench Pro
Can agents solve complex software maintenance tasks?
Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Evaluates repository-level changes; public subsets, scaffolds and reward-hacking scans are version-specific.
- Interpretation boundary
- General coding capability is enabling evidence, not direct evidence of loss of control.
SaferAI evaluates the public API without developer cooperation, combining its own benchmark runs with explicitly identified external comparison results. The report separates task capability, adversarially elicited behavior and willingness to attempt persuasion; configurations and provenance vary by benchmark.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can coding agents resolve longer, enterprise-style repository issues?
- Task population
- 1,865 problems across 41 repositories are partitioned into public, held-out and commercial subsets with human-augmented specifications and executable checks.
- Scoring and failure modes
- Pass@1 resolve rate by public, held-out or commercial subset.
- System conditions
- 1,865 tasks across 41 repositories; human-augmented specifications.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: The reported public resolve rates included 43.6% for Sonnet 4.5 and 41.8% for GPT-5 high. The best commercial-set models remained below 20%, showing subset difficulty differences.
- Validity and interpretation
- Programming-language coverage is uneven; historical repositories and human augmentations shape the measured task (§7.1).
- Appropriate research use
- Model recommendation: cite subset, scaffold and pass@1 denominator, avoiding unrestricted comparison with other SWE-bench variants.
- SWE-Bench Pro · AbstractSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
- SWE-Bench Pro · §3.3 Public, Commercial, and Held-Out SWE-Bench ProSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
- SWE-Bench Pro · §5 ResultsSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
- SWE-Bench Pro · §7.1 LimitationsSelected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established
Original source charts and materials
Protocol scope is catalogued. Exact numerical extraction and an appropriate source figure remain to be reviewed.
Related research questions
Autonomy / Sustained autonomous work →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] SWE-Bench Pro
- [2] GLM-5.2 Risk Evaluation Report. 2026-08-02. v1.1 · revised 26 August 2026 ; Report masthead and stated evaluation methodology