NUWA Evaluation dossier
SRE-Bench
Studies agentic reverse engineering with realistic binary-analysis tasks designed to reduce benchmark contamination.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Studies agentic reverse engineering with realistic binary-analysis tasks designed to reduce benchmark contamination.
- Interpretation boundary
- Reverse-engineering success is a component capability; it does not by itself establish end-to-end intrusion.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can agents recover semantics from realistic binaries without source access?
- Task population
- Nineteen privately authored programs yield 262 binary instances and 1,572 deterministic tasks; 44 protection mechanisms vary analysis difficulty.
- Scoring and failure modes
- Six-point graded instance scores, full solves and zero scores; per-model valid denominators differ.
- System conditions
- Nineteen private programs, 262 binary instances and 1,572 tasks; public/internal regimes separate.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: GPT-5.6 Sol fully solved 80/254 graded instances, versus 60/223 for Claude Fable 5.1. These denominators differ, and the study distinguishes public-facing from unconstrained internal settings (§4).
- Validity and interpretation
- Nineteen programs limit breadth; private authorship mitigates contamination but does not prove exhaustive task validity (§5).
- Appropriate research use
- Model recommendation: preserve per-model graded denominators, protection variants and safeguard condition.
- The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · AbstractSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
- The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · §3 BenchmarkSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
- The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · §4 EvaluationSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
- The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark · §5 Discussion and ConclusionSelected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] Jeremy Spence, Nicholas Assaderaghi, et al.. The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark. 2026-08-11. v2 [arXiv]; Author affiliations on the original paper title page; Figure 1