{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"exploitgym","title":"ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?","aliases":["ExploitGym"],"authors":["Zhun Wang","Nico Schiller","Hongwei Li","Srijiith Sesha Narayana","Milad Nasr","Nicholas Carlini","Xiangyu Qi","Eric Wallace","Elie Bursztein","Luca Invernizzi","Kurt Thomas","Yan Shoshitaishvili","Wenbo Guo","Jingxuan He","Thorsten Holz","Dawn Song"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"exploitgym-publication","attributionSourceId":"exploitgym-publication-author-source","originalUrl":"https://rdi.berkeley.edu/blog/exploitgym/","publicationDate":"2026-05-13","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.","zh":"主维度依据原协议的任务结果确定。主题关联见研究档案，不推断尚未审阅的问题级关联。"},"protocolIds":["exploitgym"],"contributions":[{"organizationId":"berkeley","role":"research","sourceId":"exploitgym-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"mila","role":"research","sourceId":"exploitgym-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"anthropic","role":"research","sourceId":"exploitgym-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"openai","role":"research","sourceId":"exploitgym-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"asu","role":"research","sourceId":"exploitgym-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.","zh":"在真实软件目标上评测利用构造，并核查成功是否依赖指定漏洞。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"exploitgym","title":"ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?","en":{"question":"Can agents extend supplied vulnerability triggers into concrete impact across systems?","design":"898 instances span userspace, V8 and Linux, with reproducible containers, varied protections and a time/cost-gated attempt.","findings":["The authors found substantial divergence between flag success and intended-vulnerability alignment.","GPT-5.5 and Mythos Preview aligned on only 56.7% and 69.5% of their flagged solves (§4)."],"interpretation":"NUWA interpretation: earning the success flag does not always prove use of the intended vulnerability.","limits":"Ground-truth exploits are unavailable for every task; refusal, tool failure and impossible tasks can all lower observed success (§5).","decisionRelevance":"Model recommendation: retain alignment-audit results alongside raw success, protection setting and attempt budget.","takeaway":"NUWA interpretation: earning the success flag does not always prove use of the intended vulnerability.","connections":[],"scoring":"Raw flag success and intended-vulnerability alignment are separate outcomes.","conditions":"898 instances; userspace/V8/Linux protection variants and time/cost-gated attempts.","statistics":"GPT-5.5 and Mythos Preview aligned on only 56.7% and 69.5% of their flagged solves (§4).","usage":"Model recommendation: retain alignment-audit results alongside raw success, protection setting and attempt budget."},"zh":{"question":"智能体能否跨系统将给定漏洞触发扩展为具体安全影响？","design":"898例覆盖用户态、V8和Linux，采用可复现容器、不同防护与时间成本受限尝试。","findings":["作者发现成功标志与预期漏洞归因明显分离。","GPT-5.5和Mythos Preview成功例仅56.7%、69.5%符合预期归因（§4）。"],"interpretation":"NUWA 解读：拿到成功标志不总能证明利用了指定漏洞。","limits":"并非每题都有真实利用解；拒绝、工具故障和不可解任务均可能降低成功率（§5）。","decisionRelevance":"模型建议：原始成功率旁保留归因审计、防护设置与尝试预算。","takeaway":"NUWA 解读：拿到成功标志不总能证明利用了指定漏洞。","connections":[],"scoring":"成功标志与预期漏洞归因分别报告。","conditions":"898例，用户态V8Linux保护变体及时间成本受限尝试。","statistics":"GPT-5.5和Mythos Preview成功例仅56.7%、69.5%符合预期归因（§4）。","usage":"模型建议：原始成功率旁保留归因审计、防护设置与尝试预算。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"exploitgym-publication-author-source","url":"https://arxiv.org/html/2605.11086","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established","artifactSha256":"06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e"},{"sourceId":"exploitgym-publication-author-source","url":"https://arxiv.org/html/2605.11086","locator":"§3 ExploitGym Benchmark","version":"Selected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established","artifactSha256":"06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e"},{"sourceId":"exploitgym-publication-author-source","url":"https://arxiv.org/html/2605.11086","locator":"§4 Evaluation","version":"Selected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established","artifactSha256":"06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e"},{"sourceId":"exploitgym-publication-author-source","url":"https://arxiv.org/html/2605.11086","locator":"§5 Discussion and Conclusion","version":"Selected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established","artifactSha256":"06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"22deb984bf6d43b1a2562569079cbea0e8f59471ed010a7246eee82f7cd8ebdb"}},"sources":[{"id":"exploitgym-publication","title":"ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?","url":"https://rdi.berkeley.edu/blog/exploitgym/","version":"Source capture 2026-10-25","publicationDate":"2026-05-13","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://rdi.berkeley.edu/blog/exploitgym","authors":["Zhun Wang","Nico Schiller","Hongwei Li","Srijiith Sesha Narayana","Milad Nasr","Nicholas Carlini","Xiangyu Qi","Eric Wallace","Elie Bursztein","Luca Invernizzi","Kurt Thomas","Yan Shoshitaishvili","Wenbo Guo","Jingxuan He","Thorsten Holz","Dawn Song"]}]}