{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"secbench-pro","title":"SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?","aliases":["SEC-bench Pro"],"authors":["Hwiwon Lee","Jiawei Liu","Dongjun Kim","Wubing Xia","Ziqi Zhang","Chunqiu Steven Xia","Lingming Zhang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"secbench-pro","attributionSourceId":"secbench-pro-author-source","originalUrl":"https://arxiv.org/abs/2605.26548v2","publicationDate":"2026-05-26","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Primary measure is successful task completion under a specified agent setup.","zh":"主测量是特定智能体配置下的任务完成情况。"},"protocolIds":["secbench-pro"],"contributions":[{"organizationId":"uiuc","role":"research","sourceId":"secbench-pro-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Tests long-horizon reproduction of working proof-of-concept inputs from disclosed vulnerability reports.","zh":"测试根据已披露漏洞报告进行长程任务并复现有效触发输入。"},"note":{"question":{"en":"Working proof-of-concept reproduction on a versioned software-security task set.","zh":"在有版本的软件安全任务集上复现可验证的漏洞触发输入。"},"method":{"en":"Coding agents reproducing vulnerability triggers in real software","zh":"在真实软件中复现漏洞触发条件的编程智能体"},"finding":{"en":"Tests long-horizon reproduction of working proof-of-concept inputs from disclosed vulnerability reports.","zh":"测试根据已披露漏洞报告进行长程任务并复现有效触发输入。"},"supports":{"en":"Primary measure is successful task completion under a specified agent setup.","zh":"主测量是特定智能体配置下的任务完成情况。"},"limits":{"en":"A disclosed bug and supplied instructions constrain the task; grader choices and task-set version affect scores.","zh":"任务提供已披露漏洞与触发说明；评分器及任务集版本影响结果。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"secbench-pro","title":"SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?","en":{"question":"Can agents reproduce specific disclosed vulnerabilities in large security-critical codebases?","design":"344 instances cover V8, SpiderMonkey and Linux; grading checks vulnerable, patched and latest counterparts and separates all-instance from completed-only rates.","findings":["The strongest reported model/scaffold solved 58% overall.","Success requires target attribution across the three-image check; a crash alone is insufficient (§IV-A)."],"interpretation":"NUWA interpretation: target-specific validation and timeout accounting materially define the measured capability.","limits":"Report-guided reproduction is not unguided discovery; target families and harnesses restrict external validity.","decisionRelevance":"Model recommendation: use the all-instance denominator and fixed/latest evidence before interpreting reproduction gaps.","takeaway":"NUWA interpretation: target-specific validation and timeout accounting materially define the measured capability.","connections":[],"scoring":"All-instance success under final vulnerable/fixed/latest grading; completed-only rates are sensitivity analysis.","conditions":"344 instances across browser engines and Linux; model/scaffold timeout behavior differs.","statistics":"The strongest reported model/scaffold solved 58% overall.","usage":"Model recommendation: use the all-instance denominator and fixed/latest evidence before interpreting reproduction gaps."},"zh":{"question":"智能体能否在大型安全关键代码库复现指定披露漏洞？","design":"344例覆盖V8、SpiderMonkey与Linux；判分检查漏洞、修复及最新版本，并区分全体与完成运行分母。","findings":["所报最佳模型框架整体成功58%。","成功须通过三镜像目标归因，崩溃本身不够（§IV-A）。"],"interpretation":"NUWA 解读：特定目标验证及超时记账实质定义了测量能力。","limits":"报告指引的复现不是无指引发现；目标族与框架限制外推。","decisionRelevance":"模型建议：解释复现差异前核对全体分母及修复、最新版本证据。","takeaway":"NUWA 解读：特定目标验证及超时记账实质定义了测量能力。","connections":[],"scoring":"漏洞、修复、最新三镜像最终判分下全体成功率，完成运行率仅作敏感性分析。","conditions":"344例覆盖浏览器引擎与Linux，模型框架超时不同。","statistics":"所报最佳模型框架整体成功58%。","usage":"模型建议：解释复现差异前核对全体分母及修复、最新版本证据。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"secbench-pro-author-source","url":"https://arxiv.org/html/2605.26548v2","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2605.26548v2; sha256:fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9; archived source capture; exact capture time not independently established","artifactSha256":"fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9"},{"sourceId":"secbench-pro-author-source","url":"https://arxiv.org/html/2605.26548v2","locator":"§III Design","version":"Selected original resource https://arxiv.org/html/2605.26548v2; sha256:fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9; archived source capture; exact capture time not independently established","artifactSha256":"fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9"},{"sourceId":"secbench-pro-author-source","url":"https://arxiv.org/html/2605.26548v2","locator":"§IV-A Evaluation Setup","version":"Selected original resource https://arxiv.org/html/2605.26548v2; sha256:fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9; archived source capture; exact capture time not independently established","artifactSha256":"fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9"},{"sourceId":"secbench-pro-author-source","url":"https://arxiv.org/html/2605.26548v2","locator":"§IV-F Threats to Validity","version":"Selected original resource https://arxiv.org/html/2605.26548v2; sha256:fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9; archived source capture; exact capture time not independently established","artifactSha256":"fb70724c35773ab7a7ee2c65b4396aa1004f768690b3332f4d8203b7d0929bf9"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"d7e874ef0ab12b93c1172b8346e6b6364f904a6dafad22d6feece1b303323658"}},"sources":[{"id":"secbench-pro","title":"SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?","url":"https://arxiv.org/abs/2605.26548v2","version":"arXiv:2605.26548v2","publicationDate":"2026-07-20","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-25","documentId":"arxiv:2605.26548","authors":["Hwiwon Lee ∗ Jiawei Liu \nDongjun Kim \nWubing Xia \nZiqi Zhang \nChunqiu Steven Xia \nLingming Zhang ∗"],"aliases":["https://doi.org/10.48550/arXiv.2605.26548","https://arxiv.org/abs/2605.26548","https://arxiv.org/pdf/2605.26548"]}]}