{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"sec-bench","title":"SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks","aliases":["SEC-bench"],"authors":["Hwiwon Lee","Ziqi Zhang","Hanxiao Lu","Lingming Zhang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"sec-bench-publication","attributionSourceId":"sec-bench-publication-author-source","originalUrl":"https://arxiv.org/abs/2506.11791","publicationDate":"2025-06-13","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.","zh":"主维度依据原协议的任务结果确定。主题关联见研究档案，不推断尚未审阅的问题级关联。"},"protocolIds":["sec-bench"],"contributions":[{"organizationId":"uiuc","role":"research","sourceId":"sec-bench-publication-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"purdue","role":"research","sourceId":"sec-bench-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Tests vulnerability reproduction and patching in executable software environments assembled with an automated construction pipeline.","zh":"通过自动构建的可执行软件环境，评测漏洞复现和修补能力。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"sec-bench","title":"SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks","en":{"question":"Can security-agent datasets be built with reproducible vulnerability and patch checks?","design":"An automated pipeline reconstructs C/C++ projects, verifies memory-safety defects and evaluates PoC generation and patching with multiple coding scaffolds.","findings":["The reported best success rates were 18.0% for PoC generation and 34.0% for patching on the complete dataset.","Patch tasks provide vulnerable locations, whereas PoC tasks require more exploration."],"interpretation":"NUWA interpretation: reproduction and repair are separate tasks with different information affordances.","limits":"Sanitizer-detectable memory-safety defects and C/C++ coverage exclude many web, kernel and distributed-system failure classes (§5).","decisionRelevance":"Model recommendation: retain task type and provided location information; do not equate sanitizer reproduction with exploit impact.","takeaway":"NUWA interpretation: reproduction and repair are separate tasks with different information affordances.","connections":[],"scoring":"Sanitizer-validated PoC generation success versus vulnerability-patching success.","conditions":"C/C++ projects, supported memory-safety defects and differing task information.","statistics":"The reported best success rates were 18.0% for PoC generation and 34.0% for patching on the complete dataset.","usage":"Model recommendation: retain task type and provided location information; do not equate sanitizer reproduction with exploit impact."},"zh":{"question":"能否构造带可复现漏洞和修补核验的安全智能体数据集？","design":"自动流水线重建C/C++项目，验证内存安全缺陷，以多个编程框架评测复现和修补。","findings":["完整数据集所报最佳成功率分别18.0%与34.0%。","修补提供漏洞位置，复现需更多探索。"],"interpretation":"NUWA 解读：复现与修补是信息条件不同的两类任务。","limits":"基于检测器的内存问题及C/C++覆盖排除了许多其他系统故障（§5）。","decisionRelevance":"模型建议：保留任务类型及位置提示，不把检测器复现等同于利用影响。","takeaway":"NUWA 解读：复现与修补是信息条件不同的两类任务。","connections":[],"scoring":"检测器验证的漏洞复现成功与修补成功分开。","conditions":"C/C++、支持的内存安全缺陷及不同任务提示。","statistics":"完整数据集所报最佳成功率分别18.0%与34.0%。","usage":"模型建议：保留任务类型及位置提示，不把检测器复现等同于利用影响。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"sec-bench-publication-author-source","url":"https://arxiv.org/html/2506.11791","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2506.11791; sha256:857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6; archived source capture; exact capture time not independently established","artifactSha256":"857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6"},{"sourceId":"sec-bench-publication-author-source","url":"https://arxiv.org/html/2506.11791","locator":"§3.1 Experimental Setup","version":"Selected original resource https://arxiv.org/html/2506.11791; sha256:857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6; archived source capture; exact capture time not independently established","artifactSha256":"857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6"},{"sourceId":"sec-bench-publication-author-source","url":"https://arxiv.org/html/2506.11791","locator":"§3.2 Performance of LLM Agents in Security Tasks","version":"Selected original resource https://arxiv.org/html/2506.11791; sha256:857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6; archived source capture; exact capture time not independently established","artifactSha256":"857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6"},{"sourceId":"sec-bench-publication-author-source","url":"https://arxiv.org/html/2506.11791","locator":"§5 Limitations and Future Work","version":"Selected original resource https://arxiv.org/html/2506.11791; sha256:857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6; archived source capture; exact capture time not independently established","artifactSha256":"857cf5e1b82d84f586aabf5eabc8626b00d66bd6ff321c8e170bee6eba519ca6"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"ac1aacba743dcafdd0af395cfb0745d1445081f23f932f827ed7f5b6aca85440"}},"sources":[{"id":"sec-bench-publication","title":"SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks","url":"https://arxiv.org/abs/2506.11791","version":"arXiv:2506.11791v2","publicationDate":"2025-06-13","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2506.11791","authors":["Hwiwon Lee \nZiqi Zhang \nHanxiao Lu † Lingming Zhang"],"aliases":["https://doi.org/10.48550/arXiv.2506.11791","https://arxiv.org/abs/2506.11791","https://arxiv.org/pdf/2506.11791"]}]}