Risk Evaluation · Benchmark

BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems

NUWA research note · independent review pending

First published: 2025-05-21Source checked: 2026-10-25

Original source ↗Cite this record ↓Cybersecurity →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: task information and success checks create different capability measurements.

Research question

Can one framework distinguish detection, exploitation and patching across real applications?

Design and assumptions

25 manually configured systems contain forty paid-bounty vulnerabilities; task variants change the information provided and score detect/exploit/patch separately.

Source findings

With up to three attempts, top reported scores included 12.5% Detect, 67.5% Exploit and 90% Patch from different agent setups.

These do not form a single model’s lifecycle completion rate.

NUWA interpretation

NUWA interpretation: task information and success checks create different capability measurements.

Limits and contrary evidence

Added-vulnerability coverage and incomplete invariants can miss new defects or incomplete patches; environment preparation is heavily manual.

Research and decision relevance

Model recommendation: keep task-specific agents, attempts and validation coverage instead of aggregating into one cyber score.

Precise source locations

BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems · AbstractSelected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems · §4 ExperimentsSelected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems · §6 DiscussionSelected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by Stanford University University of California, Berkeley

Authors (34)

Andy K. Zhang · Joey Ji · Celeste Menders · Riya Dulepet · Thomas Qin · Ron Y. Wang · Junrong Wu · Kyleen Liao · Jiliang Li · Jinghan Hu · Sara Hong · Nardos Demilew · Shivatmica Murgai · Jason Tran · Nishka Kacheria · Ethan Ho · Denis Liu · Lauren McLane · Olivia Bruvik · Dai-Rong Han · Seungwoo Kim · Akhil Vyas · Cuiyuanxiu Chen · Ryan Li · Weiran Xu · Jonathan Z. Ye · Prerit Choudhary · Siddharth M. Bhatia · Vikram Sivashankar · Yuxuan Bao · Dawn Song · Dan Boneh · Daniel E. Ho · Percy Liang

Evaluation protocols

BountyBench →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{bountybench,
  title = {BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems},
  url = {https://arxiv.org/abs/2505.15216},
  author = {Andy K. Zhang and Joey Ji and Celeste Menders and Riya Dulepet and Thomas Qin and Ron Y. Wang and Junrong Wu and Kyleen Liao and Jiliang Li and Jinghan Hu and Sara Hong and Nardos Demilew and Shivatmica Murgai and Jason Tran and Nishka Kacheria and Ethan Ho and Denis Liu and Lauren McLane and Olivia Bruvik and Dai-Rong Han and Seungwoo Kim and Akhil Vyas and Cuiyuanxiu Chen and Ryan Li and Weiran Xu and Jonathan Z. Ye and Prerit Choudhary and Siddharth M. Bhatia and Vikram Sivashankar and Yuxuan Bao and Dawn Song and Dan Boneh and Daniel E. Ho and Percy Liang},
  year = {2025},
  date = {2025-05-21},
  version = {arXiv:2505.15216v3},
  note = {arXiv:2505.15216v3}
}

NUWA record · BibTeX

@misc{bountybench-nuwa,
  title = {BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/bountybench/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--bountybench.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--bountybench.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Andy K. Zhang, Joey Ji, et al.. BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems. 2025-05-21. v3 [arXiv]; Author affiliations on the original paper title page; Abstract; §4 Experiments