{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"sre-bench","title":"The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark","aliases":["SRE-Bench"],"authors":["Jeremy Spence","Nicholas Assaderaghi","Feng Xiao","Jinhao Zhu","Nikil Ravi","Xiangyu Qi","Matthew Jagielski","Raluca Ada Popa","Eric Wallace","Guannan Wei","Yangruibo Ding","Zhuo Zhang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"sre-bench-publication","attributionSourceId":"sre-bench-publication-author-source","originalUrl":"https://arxiv.org/abs/2608.11469","publicationDate":"2026-08-11","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.","zh":"主维度依据原协议的任务结果确定。主题关联见研究档案，不推断尚未审阅的问题级关联。"},"protocolIds":["sre-bench"],"contributions":[{"organizationId":"columbia","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"deepmind","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"berkeley","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"vals","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"openai","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"anthropic","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"tufts","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"ucla","role":"research","sourceId":"sre-bench-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Studies agentic reverse engineering with realistic binary-analysis tasks designed to reduce benchmark contamination.","zh":"通过贴近实际的二进制分析任务研究智能体逆向工程能力，并控制评测污染。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"sre-bench","title":"The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark","en":{"question":"Can agents recover semantics from realistic binaries without source access?","design":"Nineteen privately authored programs yield 262 binary instances and 1,572 deterministic tasks; 44 protection mechanisms vary analysis difficulty.","findings":["GPT-5.6 Sol fully solved 80/254 graded instances, versus 60/223 for Claude Fable 5.1.","These denominators differ, and the study distinguishes public-facing from unconstrained internal settings (§4)."],"interpretation":"NUWA interpretation: source-code security success cannot substitute for binary-analysis measurement.","limits":"Nineteen programs limit breadth; private authorship mitigates contamination but does not prove exhaustive task validity (§5).","decisionRelevance":"Model recommendation: preserve per-model graded denominators, protection variants and safeguard condition.","takeaway":"NUWA interpretation: source-code security success cannot substitute for binary-analysis measurement.","connections":[],"scoring":"Six-point graded instance scores, full solves and zero scores; per-model valid denominators differ.","conditions":"Nineteen private programs, 262 binary instances and 1,572 tasks; public/internal regimes separate.","statistics":"GPT-5.6 Sol fully solved 80/254 graded instances, versus 60/223 for Claude Fable 5.1. These denominators differ, and the study distinguishes public-facing from unconstrained internal settings (§4).","usage":"Model recommendation: preserve per-model graded denominators, protection variants and safeguard condition."},"zh":{"question":"无源码时，智能体能否恢复真实规模二进制的语义？","design":"19个私有自编程序形成262个二进制实例、1,572项确定性任务，44种保护机制改变难度。","findings":["GPT-5.6 Sol完整解决80/254例，Claude Fable 5.1为60/223。","分母不同，且研究区分公开与内部无约束设置（§4）。"],"interpretation":"NUWA 解读：源码安全表现不能替代二进制分析测量。","limits":"19个程序限制广度；私有编写缓解污染，但不证明题目验证穷尽（§5）。","decisionRelevance":"模型建议：保留每模型有效分母、保护变体和防护条件。","takeaway":"NUWA 解读：源码安全表现不能替代二进制分析测量。","connections":[],"scoring":"六点实例分、全解及零分，逐模型有效分母不同。","conditions":"19私有程序、262二进制例、1,572任务，分公开与内部条件。","statistics":"GPT-5.6 Sol完整解决80/254例，Claude Fable 5.1为60/223。 分母不同，且研究区分公开与内部无约束设置（§4）。","usage":"模型建议：保留每模型有效分母、保护变体和防护条件。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"sre-bench-publication-author-source","url":"https://arxiv.org/html/2608.11469","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established","artifactSha256":"1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa"},{"sourceId":"sre-bench-publication-author-source","url":"https://arxiv.org/html/2608.11469","locator":"§3 Benchmark","version":"Selected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established","artifactSha256":"1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa"},{"sourceId":"sre-bench-publication-author-source","url":"https://arxiv.org/html/2608.11469","locator":"§4 Evaluation","version":"Selected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established","artifactSha256":"1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa"},{"sourceId":"sre-bench-publication-author-source","url":"https://arxiv.org/html/2608.11469","locator":"§5 Discussion and Conclusion","version":"Selected original resource https://arxiv.org/html/2608.11469; sha256:1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa; archived source capture; exact capture time not independently established","artifactSha256":"1d4839f5e2ce6a1193b841dbf689059f6a68140a440495064b4e40727076e5fa"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"79fc21bec1c10be6d20d8523ef5beb914492ea00dd095a162a724ba9feee72fb"}},"sources":[{"id":"sre-bench-publication","title":"The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark","url":"https://arxiv.org/abs/2608.11469","version":"arXiv:2608.11469v2","publicationDate":"2026-08-11","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2608.11469","authors":["Jeremy Spence","Nicholas Assaderaghi","Feng Xiao","Jinhao Zhu","Nikil Ravi","Xiangyu Qi","Matthew Jagielski","Raluca Ada Popa","Eric Wallace","Guannan Wei","Yangruibo Ding","Zhuo Zhang"],"aliases":["https://doi.org/10.48550/arXiv.2608.11469","https://arxiv.org/abs/2608.11469","https://arxiv.org/pdf/2608.11469"]}]}