Risk Evaluation · Benchmark

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

NUWA research note · independent review pending

First published: 2026-05-13Source checked: 2026-10-25

Original source ↗Cite this record ↓Cybersecurity →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: earning the success flag does not always prove use of the intended vulnerability.

Research question

Can agents extend supplied vulnerability triggers into concrete impact across systems?

Design and assumptions

898 instances span userspace, V8 and Linux, with reproducible containers, varied protections and a time/cost-gated attempt.

ExploitGym — Figure 1 — Figure 1 · Original figure 1. Read alongside the protocol scope and limitations below. Original figure and source ↗

Source findings

The authors found substantial divergence between flag success and intended-vulnerability alignment.

GPT-5.5 and Mythos Preview aligned on only 56.7% and 69.5% of their flagged solves (§4).

NUWA interpretation

NUWA interpretation: earning the success flag does not always prove use of the intended vulnerability.

Limits and contrary evidence

Ground-truth exploits are unavailable for every task; refusal, tool failure and impossible tasks can all lower observed success (§5).

Research and decision relevance

Model recommendation: retain alignment-audit results alongside raw success, protection setting and attempt budget.

Precise source locations

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · AbstractSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · §3 ExploitGym BenchmarkSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · §4 EvaluationSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · §5 Discussion and ConclusionSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by University of California, Berkeley Mila Anthropic OpenAI Arizona State University

Authors (16)

Zhun Wang · Nico Schiller · Hongwei Li · Srijiith Sesha Narayana · Milad Nasr · Nicholas Carlini · Xiangyu Qi · Eric Wallace · Elie Bursztein · Luca Invernizzi · Kurt Thomas · Yan Shoshitaishvili · Wenbo Guo · Jingxuan He · Thorsten Holz · Dawn Song

Evaluation protocols

ExploitGym →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{exploitgym,
  title = {ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?},
  url = {https://rdi.berkeley.edu/blog/exploitgym/},
  author = {Zhun Wang and Nico Schiller and Hongwei Li and Srijiith Sesha Narayana and Milad Nasr and Nicholas Carlini and Xiangyu Qi and Eric Wallace and Elie Bursztein and Luca Invernizzi and Kurt Thomas and Yan Shoshitaishvili and Wenbo Guo and Jingxuan He and Thorsten Holz and Dawn Song},
  year = {2026},
  date = {2026-05-13},
  version = {Source capture 2026-10-25},
  note = {Source capture 2026-10-25}
}

NUWA record · BibTeX

@misc{exploitgym-nuwa,
  title = {ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/exploitgym/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--exploitgym.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--exploitgym.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Zhun Wang, Nico Schiller, et al.. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?. 2026-05-13
[2] Zhun Wang, Nico Schiller, et al.. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? ; Author affiliations on the original paper title page; Abstract; §3 ExploitGym Benchmark