{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"bountybench","title":"BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems","aliases":["BountyBench"],"authors":["Andy K. Zhang","Joey Ji","Celeste Menders","Riya Dulepet","Thomas Qin","Ron Y. Wang","Junrong Wu","Kyleen Liao","Jiliang Li","Jinghan Hu","Sara Hong","Nardos Demilew","Shivatmica Murgai","Jason Tran","Nishka Kacheria","Ethan Ho","Denis Liu","Lauren McLane","Olivia Bruvik","Dai-Rong Han","Seungwoo Kim","Akhil Vyas","Cuiyuanxiu Chen","Ryan Li","Weiran Xu","Jonathan Z. Ye","Prerit Choudhary","Siddharth M. Bhatia","Vikram Sivashankar","Yuxuan Bao","Dawn Song","Dan Boneh","Daniel E. Ho","Percy Liang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"bountybench-publication","attributionSourceId":"bountybench-publication-author-source","originalUrl":"https://arxiv.org/abs/2505.15216","publicationDate":"2025-05-21","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.","zh":"主维度依据原协议的任务结果确定。主题关联见研究档案，不推断尚未审阅的问题级关联。"},"protocolIds":["bountybench"],"contributions":[{"organizationId":"stanford","role":"research","sourceId":"bountybench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"berkeley","role":"research","sourceId":"bountybench-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Evaluates detection, exploitation and patching across real-world codebases, with task outcomes grounded in vulnerability lifecycles.","zh":"围绕真实代码库中的漏洞生命周期，分别评测发现、利用和修补任务。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"bountybench","title":"BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems","en":{"question":"Can one framework distinguish detection, exploitation and patching across real applications?","design":"25 manually configured systems contain forty paid-bounty vulnerabilities; task variants change the information provided and score detect/exploit/patch separately.","findings":["With up to three attempts, top reported scores included 12.5% Detect, 67.5% Exploit and 90% Patch from different agent setups.","These do not form a single model’s lifecycle completion rate."],"interpretation":"NUWA interpretation: task information and success checks create different capability measurements.","limits":"Added-vulnerability coverage and incomplete invariants can miss new defects or incomplete patches; environment preparation is heavily manual.","decisionRelevance":"Model recommendation: keep task-specific agents, attempts and validation coverage instead of aggregating into one cyber score.","takeaway":"NUWA interpretation: task information and success checks create different capability measurements.","connections":[],"scoring":"Detect, Exploit and Patch pass criteria differ; reported best-of-up-to-three attempts.","conditions":"25 systems and forty bounties with task-specific information affordances.","statistics":"With up to three attempts, top reported scores included 12.5% Detect, 67.5% Exploit and 90% Patch from different agent setups.","usage":"Model recommendation: keep task-specific agents, attempts and validation coverage instead of aggregating into one cyber score."},"zh":{"question":"同一框架能否区分真实应用中的发现、利用与修补？","design":"25个手工配置系统含40项获赏漏洞，通过不同提示信息分别评测发现、利用和修补。","findings":["最多三次尝试时，不同框架最佳分别为12.5%、67.5%、90%。","这不是同一模型的全生命周期完成率。"],"interpretation":"NUWA 解读：提供信息与成功检查定义不同能力测量。","limits":"已添加漏洞覆盖和不完整不变量可能漏新问题或未充分修复，环境构造很依赖人工。","decisionRelevance":"模型建议：保留各任务智能体、尝试数及验证覆盖，不合成单一网络分。","takeaway":"NUWA 解读：提供信息与成功检查定义不同能力测量。","connections":[],"scoring":"发现、利用、修补通过标准不同，所报最多三次最佳。","conditions":"25系统40漏洞，各任务提供信息不同。","statistics":"最多三次尝试时，不同框架最佳分别为12.5%、67.5%、90%。","usage":"模型建议：保留各任务智能体、尝试数及验证覆盖，不合成单一网络分。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"bountybench-publication-author-source","url":"https://arxiv.org/html/2505.15216","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established","artifactSha256":"20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52"},{"sourceId":"bountybench-publication-author-source","url":"https://arxiv.org/html/2505.15216","locator":"§4 Experiments","version":"Selected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established","artifactSha256":"20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52"},{"sourceId":"bountybench-publication-author-source","url":"https://arxiv.org/html/2505.15216","locator":"§6 Discussion","version":"Selected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established","artifactSha256":"20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"b6416111df0a9760bf5e76fad0e8f51ac8fe152475956136b9e9cff21e406851"}},"sources":[{"id":"bountybench-publication","title":"BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems","url":"https://arxiv.org/abs/2505.15216","version":"arXiv:2505.15216v3","publicationDate":"2025-05-21","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2505.15216","authors":["Andy K. Zhang","Joey Ji","Celeste Menders","Riya Dulepet","Thomas Qin","Ron Y. Wang","Junrong Wu","Kyleen Liao","Jiliang Li","Jinghan Hu","Sara Hong","Nardos Demilew","Shivatmica Murgai","Jason Tran","Nishka Kacheria","Ethan Ho","Denis Liu","Lauren McLane","Olivia Bruvik","Dai-Rong Han","Seungwoo Kim","Akhil Vyas","Cuiyuanxiu Chen","Ryan Li","Weiran Xu","Jonathan Z. Ye","Prerit Choudhary","Siddharth M. Bhatia","Vikram Sivashankar","Yuxuan Bao","Dawn Song","Dan Boneh","Daniel E. Ho","Percy Liang"],"aliases":["https://doi.org/10.48550/arXiv.2505.15216","https://arxiv.org/abs/2505.15216","https://arxiv.org/pdf/2505.15216"]}]}