Risk Evaluation · Benchmark

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

NUWA research note · independent review pending

First published: 2026-08-11Source checked: 2026-10-25

Original source ↗Cite this record ↓Cybersecurity →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: source-code security success cannot substitute for binary-analysis measurement.

Research question

Can agents recover semantics from realistic binaries without source access?

Design and assumptions

Nineteen privately authored programs yield 262 binary instances and 1,572 deterministic tasks; 44 protection mechanisms vary analysis difficulty.

SRE-Bench — Figure 1 — Figure 1 · Original figure 1. Read alongside the protocol scope and limitations below. Original figure and source ↗

Source findings

GPT-5.6 Sol fully solved 80/254 graded instances, versus 60/223 for Claude Fable 5.1.

These denominators differ, and the study distinguishes public-facing from unconstrained internal settings (§4).

NUWA interpretation

NUWA interpretation: source-code security success cannot substitute for binary-analysis measurement.

Limits and contrary evidence

Nineteen programs limit breadth; private authorship mitigates contamination but does not prove exhaustive task validity (§5).

Research and decision relevance

Model recommendation: preserve per-model graded denominators, protection variants and safeguard condition.

Precise source locations

The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · AbstractSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · §3 BenchmarkSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · §4 EvaluationSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · §5 Discussion and ConclusionSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by Columbia University Google DeepMind University of California, Berkeley Vals AI OpenAI Anthropic Tufts University University of California, Los Angeles

Authors (12)

Jeremy Spence · Nicholas Assaderaghi · Feng Xiao · Jinhao Zhu · Nikil Ravi · Xiangyu Qi · Matthew Jagielski · Raluca Ada Popa · Eric Wallace · Guannan Wei · Yangruibo Ding · Zhuo Zhang

Evaluation protocols

SRE-Bench →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{sre-bench,
  title = {The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark},
  url = {https://arxiv.org/abs/2608.11469},
  author = {Jeremy Spence and Nicholas Assaderaghi and Feng Xiao and Jinhao Zhu and Nikil Ravi and Xiangyu Qi and Matthew Jagielski and Raluca Ada Popa and Eric Wallace and Guannan Wei and Yangruibo Ding and Zhuo Zhang},
  year = {2026},
  date = {2026-08-11},
  version = {arXiv:2608.11469v2},
  note = {arXiv:2608.11469v2}
}

NUWA record · BibTeX

@misc{sre-bench-nuwa,
  title = {The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/sre-bench/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--sre-bench.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--sre-bench.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Jeremy Spence, Nicholas Assaderaghi, et al.. The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark. 2026-08-11. v2 [arXiv]; Author affiliations on the original paper title page; Abstract; §3 Benchmark