{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"cybench","title":"Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models","aliases":["Cybench"],"authors":["Andy K. Zhang","Neil Perry","Riya Dulepet","Joey Ji","Celeste Menders","Justin W. Lin","Eliot Jones","Gashon Hussein","Samantha Liu","Donovan Jasper","Pura Peetathawatchai","Ari Glenn","Vikram Sivashankar","Daniel Zamoshchin","Leo Glikbarg","Derek Askaryar","Mike Yang","Teddy Zhang","Rishi Alluri","Nathan Tran","Rinnara Sangpisit","Polycarpos Yiorkadjis","Kenny Osele","Gautham Raghupathi","Dan Boneh","Daniel E. Ho","Percy Liang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"cybench","attributionSourceId":"cybench-author-source","originalUrl":"https://arxiv.org/abs/2408.08926v4","publicationDate":"2024-08-15","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.","zh":"主维度依据来源实际测量的构念；具体指标可能属于其他维度。"},"protocolIds":["cybench"],"contributions":[{"organizationId":"stanford","role":"research","sourceId":"cybench-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Cyber capability on competition tasks with optional subtask guidance.","zh":"竞赛任务及可选子任务提示下的网络能力。"},"note":{"question":{"en":"Task completion on 40 professional CTF problems.","zh":"40 项专业 CTF 任务的完成情况。"},"method":{"en":"AI agent solving CTF tasks","zh":"完成 CTF 任务的 AI 智能体"},"finding":{"en":"Cyber capability on competition tasks with optional subtask guidance.","zh":"竞赛任务及可选子任务提示下的网络能力。"},"supports":null,"limits":{"en":"Competition tasks and supplied guidance differ from operational attacks.","zh":"竞赛任务与提供的提示不同于现实攻击条件。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"cybench","title":"Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models","en":{"takeaway":"Cybench separates finishing a CTF task without help from finishing after stepwise guidance. Partial subtask performance is a third endpoint and cannot be substituted for either.","question":"How capable are language-model agents at multi-step security tasks, and which steps remain difficult when intermediate goals are supplied?","design":"CTF tasks provide a description, local or remote starter material and an evaluator in containerized environments. Tasks span security categories and first-solve times. In unguided mode the agent receives the overall goal; in subtask mode it receives sequential intermediate questions and has a limited attempt budget for each. The baseline agent follows an action–execution–memory-update loop with a short recent history.","scoring":"Unguided success and final guided success are binary task endpoints; subtask performance is the fraction of intermediate questions solved, macro-averaged across tasks. Table 2 gives single attempts; Table 3 uses the maximum of multiple attempts. A unique success marker or correct submitted answer determines completion.","findings":["Different models lead on different endpoints, so the source itself does not support a single ordering across unguided, guided and intermediate-task performance (§5.1).","The public MOTP example decomposes a web task into identifying relevant files, recognizing a condition and finishing the challenge. Intermediate answers make progress visible without guaranteeing independent completion (§§2.2–2.3, Table 1).","Scaffold variants and expanded memory change experimental conditions; the paper explicitly warns that stronger external evaluations use different iteration limits and scaffolds (Appendix I.1)."],"interpretation":"Guidance measures the ability to execute supplied steps as well as independent discovery. First-solve time provides a difficulty proxy; it is not an individual human baseline directly comparable to METR task duration.","limits":"Competition problems are intentionally solvable, typically short and small, and may not represent accidental vulnerabilities in large systems. Competition timing differs across events. Short memory, limited iterations and few repetitions constrain the baseline. Later Cybench subsets or harnesses require separate version labels.","connections":["The source relates CTF challenges to cybersecurity capability evaluation. CyberGym substitutes real-project reproduction with a patched control, while AgentCyberRange changes the target to stateful range progression; no common denominator is established."],"decisionRelevance":"Model recommendation: select the endpoint before comparing systems, display guidance and attempt rules, and avoid translating CTF completion into a probability of real-world compromise.","statistics":"Table 2 is a single-attempt evaluation; Table 3 takes the best of three attempts. Subtask metrics are macro-averaged, not a pooled count of all questions. Unreported interval endpoints remain unknown."},"zh":{"takeaway":"Cybench 区分无指导完成 CTF 与逐步提示后的完成，中间子任务表现又是第三种终点，不能相互替代。","question":"语言模型智能体能够完成怎样的多步安全任务？提供中间目标后，哪些步骤仍困难？","design":"CTF 任务在容器环境中提供说明、本地或远程材料及验收器，覆盖多类问题和首次解出时间。无指导模式只给总目标；子任务模式逐步提供中间问题，每步有有限尝试预算。基线智能体以行动、执行、更新记忆的循环工作，只保留较短近期历史。","scoring":"无指导完成与最终指导完成均为二元终点；子任务表现是中间问题成功比例，按任务宏平均。表 2 是单次尝试，表 3 取多次尝试最大值。完成依赖正确提交或唯一成功标记。","findings":["不同模型在不同终点领先，来源本身不支持跨三种指标的唯一排序（§5.1）。","公开 MOTP 示例把任务拆成识别相关文件、理解条件和完成挑战；中间答案可展示进展，却不保证独立完成（§§2.2–2.3，表 1）。","脚手架和扩展记忆改变实验条件；附录 I.1 明确提醒，外部较强评测使用了不同迭代上限与脚手架。"],"interpretation":"指导模式同时测量执行既定步骤与自主发现。首次解出时间只是难度代理，不能直接当作 METR 式个人人类用时基线。","limits":"竞赛题刻意设计为可解，通常较短且代码量小，不代表大型系统中的意外漏洞。不同竞赛计时存在差别；有限记忆、迭代和重复限制了基线。后续子集与 Harness 需要独立版本。","connections":["论文将 CTF 与网络能力评估联系起来。CyberGym 改为带修复对照的真实项目复现，AgentCyberRange 则改变为有状态的靶场推进，三者未建立共同分母。"],"decisionRelevance":"模型建议：先确定终点，再比较系统，并显示指导方式和尝试规则，不把 CTF 完成率换算为现实攻陷概率。","statistics":"表 2 是单次尝试，表 3 取三次最佳；子任务采用宏平均，并非所有问题的合并比例。未披露的区间端点保持未知。"},"figureRefs":[],"assertionPolicy":{"design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","connections":"nuwa-interpretation-with-source-refs","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"cybench-author-source","url":"https://arxiv.org/html/2408.08926v4","locator":"§2.1 Task Specification","version":"arXiv:2408.08926v4"},{"sourceId":"cybench-author-source","url":"https://arxiv.org/html/2408.08926v4","locator":"§2.2 Task Example: MOTP; §2.3 Subtasks; Table 1","version":"arXiv:2408.08926v4"},{"sourceId":"cybench-author-source","url":"https://arxiv.org/html/2408.08926v4","locator":"§2.4 Metrics","version":"arXiv:2408.08926v4"},{"sourceId":"cybench-author-source","url":"https://arxiv.org/html/2408.08926v4","locator":"§3.3 Task Selection; Appendix F First Solve Time","version":"arXiv:2408.08926v4"},{"sourceId":"cybench-author-source","url":"https://arxiv.org/html/2408.08926v4","locator":"§4 LM-based Agent; §5.1 Model Capabilities; Tables 2–3","version":"arXiv:2408.08926v4"},{"sourceId":"cybench-author-source","url":"https://arxiv.org/html/2408.08926v4","locator":"Appendix I.1 Limited agent scaffolding; I.2 Limitation of data sources","version":"arXiv:2408.08926v4"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":"full-text","readingCoverage":"selected-methods-results-limitations-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"84bee6dd7e8b88728cbb5b2d2b3a12224c75634902ec59a868610603e90818a1"}},"sources":[{"id":"cybench","title":"Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models","url":"https://arxiv.org/abs/2408.08926v4","version":"arXiv:2408.08926v4","publicationDate":"2025-04-12","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"arxiv:2408.08926","authors":["Andy K. Zhang","Neil Perry","Riya Dulepet","Joey Ji","Celeste Menders","Justin W. Lin","Eliot Jones"],"aliases":["https://doi.org/10.48550/arXiv.2408.08926","https://arxiv.org/abs/2408.08926","https://arxiv.org/pdf/2408.08926"]}]}