{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"cybergym","title":"CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale","aliases":["CyberGym"],"authors":["Zhun Wang","Tianneng Shi","Jingxuan He","Matthew Cai","Jialin Zhang","Dawn Song"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"cybergym","attributionSourceId":"cybergym-author-source","originalUrl":"https://arxiv.org/abs/2506.02548v3","publicationDate":"2025-06-03","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.","zh":"主维度依据来源实际测量的构念；具体指标可能属于其他维度。"},"protocolIds":["cybergym"],"contributions":[{"organizationId":"berkeley","role":"research","sourceId":"cybergym-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Vulnerability reproduction in real software projects.","zh":"真实软件项目中的漏洞复现。"},"note":{"question":{"en":"Validated reproduction across a corpus of historical vulnerabilities.","zh":"历史漏洞集合中的经验证复现表现。"},"method":{"en":"AI agent reproducing historical vulnerabilities","zh":"复现历史漏洞的 AI 智能体"},"finding":{"en":"Vulnerability reproduction in real software projects.","zh":"真实软件项目中的漏洞复现。"},"supports":null,"limits":{"en":"Vulnerability reproduction is narrower than autonomous compromise of deployed systems.","zh":"漏洞复现的范围小于对部署系统的自主入侵。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"cybergym","title":"CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale","en":{"takeaway":"CyberGym tests whether an agent can turn a vulnerability description and source tree into an execution-verified reproducer. A pre-patch crash alone does not satisfy its primary success criterion.","question":"Can agents reproduce real historical software vulnerabilities at scale, and how do input information, model and agent configuration change that ability?","design":"Historical OSS-Fuzz cases are filtered for informative descriptions, reproducibility and redundancy. Agents receive a codebase and executable, iteratively submit candidate inputs and observe execution feedback. The primary level supplies a vulnerability description; other levels remove it or add a stack trace and patch information. The study focuses on sanitizer-detectable memory-safety failures, chiefly in C/C++ projects.","scoring":"A successful reproducer triggers a sanitizer failure in the vulnerable version and no sanitizer failure in the patched version. Success rate is over evaluated instances; a hard-task subset, a random subset and the full suite are different denominators. Agent feedback and final receipt must be distinguished.","findings":["The difficulty ladder demonstrates that richer diagnostic information makes reproduction easier for the examined configuration (§4, Figure 6).","Pre- and post-training-cutoff splits do not show a statistically significant success-rate difference for the examined models (Table 2). This is limited evidence against one contamination effect, not proof of an uncontaminated benchmark.","The patched-version control helps distinguish a target-specific reproducer from an unrelated crash; the authors separately investigate incomplete patches and new vulnerabilities (§§3.2, 5)."],"interpretation":"Execution-based validation gives the score a concrete meaning. Reproduction capability nevertheless differs from reliable exploitation, choosing malicious goals or progressing through a network. New-vulnerability investigations are additional experiments rather than replacements for the primary benchmark endpoint.","limits":"Coverage favors sanitizer-visible memory safety and the available OSS-Fuzz/ARVO corpus. Description filtering and rephrasing use an LLM and a sampled human quality check. Public code and reports create contamination uncertainty. More input information changes the task. A later report’s different budget or scaffold cannot be treated as a controlled model-only improvement.","connections":["ARVO supplies much of the reproducible case infrastructure (§3.3). Cybench uses competition tasks and flags; AgentCyberRange examines state-dependent progression through a range. These are complementary constructs, not interchangeable score scales."],"decisionRelevance":"Model recommendation: require both vulnerable-trigger and patched-clean receipts, identify the exact task subset and information level, and keep run budget and agent settings alongside each score.","statistics":"§4 Table 2 retains success numerators/denominators and Fisher exact / two-proportion Z tests for cutoff splits. Non-significance is not equivalence. Other result-specific intervals remain in their source records."},"zh":{"takeaway":"CyberGym 检验智能体能否把漏洞描述与代码库转化为通过执行验证的复现输入；仅让修复前版本崩溃不足以满足主要成功标准。","question":"智能体能否规模化复现历史真实漏洞？提供的信息、模型和智能体配置如何改变这一能力？","design":"历史 OSS-Fuzz 案例经描述质量、可复现性及去重过滤。智能体获得代码与可执行文件，提交候选输入并利用执行反馈迭代。主要难度提供漏洞描述，其他难度去除描述或增加栈轨迹和补丁。范围主要是 C/C++ 项目中可由 sanitizer 检出的内存安全问题。","scoring":"成功须同时满足漏洞版本触发 sanitizer 错误、修复版本不触发 sanitizer 错误。成功率的分母是实际评测实例；困难子集、随机子集与全集不同。过程反馈不能等同于最终验收。","findings":["难度阶梯显示，在所研究配置中，更多诊断信息使复现更容易（§4，图 6）。","训练截止日前后分组未发现所考察模型成功率的显著差异（表 2）；这只是针对一种污染效应的有限证据，不能证明基准无污染。","修复版本对照有助于区分目标漏洞与无关崩溃；作者另行研究不完整补丁与新漏洞（§§3.2、5）。"],"interpretation":"执行验证使分数含义具体，但漏洞复现与可靠利用、自主选择恶意目标、推进网络攻击链不同；新漏洞调查是额外实验，不替代主要基准终点。","limits":"覆盖偏向 sanitizer 可见的内存安全与可得的 OSS-Fuzz/ARVO 数据。描述过滤及改写使用模型，并以抽样人工检查评估质量。公开代码和报告带来污染不确定性，更多提示会改变任务；后续报告改变预算或脚手架不能视为仅模型变化的受控提升。","connections":["ARVO 提供大量可复现案例基础设施（§3.3）。Cybench 使用竞赛任务与旗标，AgentCyberRange 研究靶场中有状态依赖的推进；这些构念互补，分数不能互换。"],"decisionRelevance":"模型建议：同时保留漏洞触发与修复干净验收，标明任务子集、信息难度、预算和智能体设置。","statistics":"§4 表 2 保留截止日期分组的成功分子、分母及 Fisher 精确检验、双比例 Z 检验；不显著不等于等效，其他区间需随各来源记录保留。"},"figureRefs":[{"figureId":"cybergym-figure","sourceId":"cybergym-figure-source","locator":"Figure 2","original":"/images/nuwa-index/cybergym-original.png","provenanceUrl":"https://arxiv.org/html/2506.02548v3/x1.png","disposition":"existing-repository-provenance-not-newly-authorized","review":"Retain existing repository provenance only. Image suitability and rights are not newly approved by this editorial draft."}],"assertionPolicy":{"design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","connections":"nuwa-interpretation-with-source-refs","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"cybergym-author-source","url":"https://arxiv.org/html/2506.02548v3","locator":"§3.1 Scope, Rationale, and Limitations","version":"arXiv:2506.02548v3"},{"sourceId":"cybergym-author-source","url":"https://arxiv.org/html/2506.02548v3","locator":"§3.2 Task Input and Output; Execution-Based Evaluation Metrics; Different Levels of Difficulty","version":"arXiv:2506.02548v3"},{"sourceId":"cybergym-author-source","url":"https://arxiv.org/html/2506.02548v3","locator":"§3.3 Quality Assurance; Benchmark Scale and Diversity; Table 3","version":"arXiv:2506.02548v3"},{"sourceId":"cybergym-author-source","url":"https://arxiv.org/html/2506.02548v3","locator":"§4 Limited Impact of Potential Data Contamination; Table 2","version":"arXiv:2506.02548v3"},{"sourceId":"cybergym-author-source","url":"https://arxiv.org/html/2506.02548v3","locator":"§4 Richer Input Information Enhances Reproduction Effort; Figure 6","version":"arXiv:2506.02548v3"},{"sourceId":"cybergym-author-source","url":"https://arxiv.org/html/2506.02548v3","locator":"§5 From Benchmarking to Direct Security Impact; Ethics Statement","version":"arXiv:2506.02548v3"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":"full-text","readingCoverage":"selected-methods-results-limitations-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"feb7754fb7d42a74c497b769ccfdf905fe4b5610ffc889f2de8d9b2db9cc8ab5"}},"sources":[{"id":"cybergym","title":"CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale","url":"https://arxiv.org/abs/2506.02548v3","version":"arXiv:2506.02548v3","publicationDate":"2026-03-24","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"arxiv:2506.02548","authors":["Zhun Wang","Tianneng Shi 1 1 footnotemark: 1","Jingxuan He","Matthew Cai","Jialin Zhang","Dawn Song","{ zhun.wang","stneng }@berkeley.edu"],"aliases":["https://doi.org/10.48550/arXiv.2506.02548","https://arxiv.org/abs/2506.02548","https://arxiv.org/pdf/2506.02548"]}]}