{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"evmbench","title":"EVMbench","aliases":["EVMbench"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"evmbench-primary","attributionSourceId":"evmbench-primary","originalUrl":"https://openai.com/index/introducing-evmbench/","publicationDate":null,"updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["evmbench"],"contributions":[],"summary":{"en":"Can agents detect, repair and exploit smart-contract vulnerabilities in the benchmark?","zh":"智能体能否在基准中检测、修补或利用智能合约漏洞？"},"note":{"question":{"en":"Can agents detect, repair and exploit smart-contract vulnerabilities in the benchmark?","zh":"智能体能否在基准中检测、修补或利用智能合约漏洞？"},"method":{"en":"Separates detection, patching and exploitation modes with their stated task sets.","zh":"分别组织检测、修补和利用模式及对应任务集。"},"finding":null,"supports":null,"limits":{"en":"Modes and task subsets cannot be combined into an undifferentiated cyber-risk score.","zh":"不同模式和子集不能合并成无区分的网络风险分数。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"evmbench","title":"EVMbench","en":{"question":"Can agents detect, patch and exploit historical smart-contract vulnerabilities?","design":"117 curated vulnerabilities from forty audits support separate modes; isolated-chain execution and deterministic transaction replay validate exploit tasks.","findings":["GPT-5.3-Codex via Codex CLI scored 71.0% in exploit mode, versus GPT-5’s 33.3%."],"interpretation":"NUWA interpretation: detect, patch and exploit scores measure distinct capabilities.","limits":"Known-ground-truth detection misses novel issues; sequential replay excludes timing-dependent behavior, and local single-chain tests limit realism.","decisionRelevance":"Model recommendation: cite mode and harness conditions; do not treat exploit scores as live-network compromise rates.","takeaway":"NUWA interpretation: detect, patch and exploit scores measure distinct capabilities.","connections":[],"scoring":"Separate detection recall, patch validation and programmatically verified exploit score.","conditions":"117 vulnerabilities from forty audits; isolated local single-chain replay.","statistics":"GPT-5.3-Codex via Codex CLI scored 71.0% in exploit mode, versus GPT-5’s 33.3%.","usage":"Model recommendation: cite mode and harness conditions; do not treat exploit scores as live-network compromise rates."},"zh":{"question":"智能体能否发现、修补及利用历史智能合约漏洞？","design":"40次审计精选117个漏洞，分三模式，以隔离链及确定性交易重放验证。","findings":["Codex CLI的GPT-5.3-Codex利用得分71.0%，GPT-5为33.3%。"],"interpretation":"NUWA 解读：发现、修补及利用测不同能力。","limits":"已知真值会漏新问题，顺序重放不覆盖时序行为，本地单链限制真实度。","decisionRelevance":"模型建议：引用模式和框架条件，不当真实网络失陷率。","takeaway":"NUWA 解读：发现、修补及利用测不同能力。","connections":[],"scoring":"分别测发现召回、修补验证与程序验证利用得分。","conditions":"40审计117漏洞，本地隔离单链重放。","statistics":"Codex CLI的GPT-5.3-Codex利用得分71.0%，GPT-5为33.3%。","usage":"模型建议：引用模式和框架条件，不当真实网络失陷率。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"evmbench-primary","url":"https://openai.com/index/introducing-evmbench/","locator":"Introducing EVMbench: capability modes (official article paragraphs 31–44)","version":"Selected original resource https://openai.com/index/introducing-evmbench/; sha256:0d7b334d1452cbe75c15509ed172d187c3c9e70ce186f39678761a9e9ac8ddaa; web-tool-extracted-response; not original HTML","artifactSha256":"0d7b334d1452cbe75c15509ed172d187c3c9e70ce186f39678761a9e9ac8ddaa"},{"sourceId":"evmbench-primary","url":"https://openai.com/index/introducing-evmbench/","locator":"Limitations (paragraphs 45–49)","version":"Selected original resource https://openai.com/index/introducing-evmbench/; sha256:0d7b334d1452cbe75c15509ed172d187c3c9e70ce186f39678761a9e9ac8ddaa; web-tool-extracted-response; not original HTML","artifactSha256":"0d7b334d1452cbe75c15509ed172d187c3c9e70ce186f39678761a9e9ac8ddaa"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"3c3716d20ad424aff8ebd1419e57793140eebc3048700370d05b76416c266956"}},"sources":[{"id":"evmbench-primary","title":"EVMbench","url":"https://openai.com/index/introducing-evmbench/","version":"2026-10-25 source capture","publicationDate":null,"datePrecision":"unknown","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"https://openai.com/index/introducing-evmbench"}]}