{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"swe-bench-pro","title":"SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?","aliases":["SWE-Bench Pro"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"swe-bench-pro-primary","attributionSourceId":"swe-bench-pro-primary","originalUrl":"https://arxiv.org/abs/2509.16941","publicationDate":null,"updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["swe-bench-pro"],"contributions":[],"summary":{"en":"Can agents solve complex software maintenance tasks?","zh":"智能体能否解决复杂软件维护任务？"},"note":{"question":{"en":"Can agents solve complex software maintenance tasks?","zh":"智能体能否解决复杂软件维护任务？"},"method":{"en":"Evaluates repository-level changes; public subsets, scaffolds and reward-hacking scans are version-specific.","zh":"评测仓库级代码修改；公开子集、脚手架和奖励投机扫描有各自版本。"},"finding":null,"supports":null,"limits":{"en":"General coding capability is enabling evidence, not direct evidence of loss of control.","zh":"一般编程能力属于使能证据，不是失控的直接证据。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"swe-bench-pro","title":"SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?","en":{"question":"Can coding agents resolve longer, enterprise-style repository issues?","design":"1,865 problems across 41 repositories are partitioned into public, held-out and commercial subsets with human-augmented specifications and executable checks.","findings":["The reported public resolve rates included 43.6% for Sonnet 4.5 and 41.8% for GPT-5 high.","The best commercial-set models remained below 20%, showing subset difficulty differences."],"interpretation":"NUWA interpretation: public and proprietary enterprise subsets must remain distinct populations.","limits":"Programming-language coverage is uneven; historical repositories and human augmentations shape the measured task (§7.1).","decisionRelevance":"Model recommendation: cite subset, scaffold and pass@1 denominator, avoiding unrestricted comparison with other SWE-bench variants.","takeaway":"NUWA interpretation: public and proprietary enterprise subsets must remain distinct populations.","connections":[],"scoring":"Pass@1 resolve rate by public, held-out or commercial subset.","conditions":"1,865 tasks across 41 repositories; human-augmented specifications.","statistics":"The reported public resolve rates included 43.6% for Sonnet 4.5 and 41.8% for GPT-5 high. The best commercial-set models remained below 20%, showing subset difficulty differences.","usage":"Model recommendation: cite subset, scaffold and pass@1 denominator, avoiding unrestricted comparison with other SWE-bench variants."},"zh":{"question":"编程智能体能否解决更长的企业式仓库问题？","design":"41仓库1,865问题分公开、保留与商业子集，配人工增强规格和可执行检查。","findings":["所报公开解决率Sonnet 4.5为43.6%，GPT-5 high为41.8%。","商业集最佳仍低于20%，子集难度明显不同。"],"interpretation":"NUWA 解读：公开与专有企业集必须保持不同总体。","limits":"语言覆盖不均，历史仓库与人工增强影响任务定义（§7.1）。","decisionRelevance":"模型建议：引用子集、框架与pass@1分母，不无条件对比其他SWE-bench版本。","takeaway":"NUWA 解读：公开与专有企业集必须保持不同总体。","connections":[],"scoring":"按公开、保留、商业子集计pass@1解决率。","conditions":"41仓库1,865任务，人工增强规格。","statistics":"所报公开解决率Sonnet 4.5为43.6%，GPT-5 high为41.8%。 商业集最佳仍低于20%，子集难度明显不同。","usage":"模型建议：引用子集、框架与pass@1分母，不无条件对比其他SWE-bench版本。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"swe-bench-pro-primary","url":"https://arxiv.org/html/2509.16941","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established","artifactSha256":"5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d"},{"sourceId":"swe-bench-pro-primary","url":"https://arxiv.org/html/2509.16941","locator":"§3.3 Public, Commercial, and Held-Out SWE-Bench Pro","version":"Selected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established","artifactSha256":"5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d"},{"sourceId":"swe-bench-pro-primary","url":"https://arxiv.org/html/2509.16941","locator":"§5 Results","version":"Selected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established","artifactSha256":"5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d"},{"sourceId":"swe-bench-pro-primary","url":"https://arxiv.org/html/2509.16941","locator":"§7.1 Limitations","version":"Selected original resource https://arxiv.org/html/2509.16941; sha256:5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d; archived source capture; exact capture time not independently established","artifactSha256":"5694b2eb822d6fb6ea23c29191cf4c29e3357d0b0547b362e7393d1ef0e4919d"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"dd8ff0d26c9da84e17049d9047ca2559dedd4bb0d350de0ce6ce9b71f08a366b"}},"sources":[{"id":"swe-bench-pro-primary","title":"SWE-Bench Pro","url":"https://arxiv.org/abs/2509.16941","version":"2026-10-25 source capture","publicationDate":null,"datePrecision":"unknown","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"arxiv:2509.16941"}]}