{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"catastrophic-cyber-benchmark","title":"Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities","aliases":[],"authors":["Andrey Anurin","Jonathan Ng","Kibo Schaffer","Jason Schreiber","Esben Kran"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"catastrophic-cyber-benchmark-publication","attributionSourceId":"catastrophic-cyber-benchmark-publication-author-source","originalUrl":"https://arxiv.org/abs/2410.09114","publicationDate":"2024-10-10","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":["catastrophic-cyber-benchmark"],"contributions":[{"organizationId":"apart","role":"research","sourceId":"three-cb-attribution","locator":"Original institutional announcement: Apart Research paper"}],"summary":{"en":"Evaluates agent cyber-offense capabilities with executable tasks and examines robustness across prompting and task conditions.","zh":"使用可执行任务评测智能体网络攻击能力，并研究提示和任务条件下的稳健性。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"catastrophic-cyber-benchmark","title":"Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities","en":{"question":"How sensitive are cyber-capability evaluations to elicitation and communication protocol?","design":"3CB evaluates fourteen models on fifteen challenges and eighty elicitation configurations, with four ethically withheld challenges retained privately.","findings":["GPT-4o’s completion improved by 24.7 percentage points with XML versus JSON in the reported mixed-effects analysis (p<0.001).","Other models showed different protocol sensitivity (§3.2)."],"interpretation":"NUWA interpretation: output-interface choice can confound apparent cyber capability.","limits":"Challenge completion does not measure catastrophic outcomes; withheld tasks and finite challenge breadth limit public reproduction.","decisionRelevance":"Model recommendation: record protocol and elicitation configuration before interpreting model rankings or risk labels.","takeaway":"NUWA interpretation: output-interface choice can confound apparent cyber capability.","connections":[],"scoring":"Challenge completion under alternative elicitation protocols; not catastrophic-harm probability.","conditions":"Fourteen models, fifteen challenges, eighty elicitation configurations.","statistics":"GPT-4o’s completion improved by 24.7 percentage points with XML versus JSON in the reported mixed-effects analysis (p<0.001). Other models showed different protocol sensitivity (§3.2).","usage":"Model recommendation: record protocol and elicitation configuration before interpreting model rankings or risk labels."},"zh":{"question":"网络能力评价对激发及通信格式有多敏感？","design":"3CB以15题、80种激发配置评价14模型，另有四题出于伦理原因保留私有。","findings":["所报混合效应分析中GPT-4o用XML较JSON提高24.7个百分点，p<0.001。","不同模型格式敏感性有别（§3.2）。"],"interpretation":"NUWA 解读：输出接口可混杂表面网络能力。","limits":"题目完成不测灾难结果，保留任务与有限广度限制公开复现。","decisionRelevance":"模型建议：解释排名或风险标签前记录格式及激发配置。","takeaway":"NUWA 解读：输出接口可混杂表面网络能力。","connections":[],"scoring":"不同激发格式下题目完成，不是灾难伤害概率。","conditions":"14模型15题80激发配置。","statistics":"所报混合效应分析中GPT-4o用XML较JSON提高24.7个百分点，p<0.001。 不同模型格式敏感性有别（§3.2）。","usage":"模型建议：解释排名或风险标签前记录格式及激发配置。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"catastrophic-cyber-benchmark-publication-author-source","url":"https://arxiv.org/html/2410.09114","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2410.09114; sha256:b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f; archived source capture; exact capture time not independently established","artifactSha256":"b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f"},{"sourceId":"catastrophic-cyber-benchmark-publication-author-source","url":"https://arxiv.org/html/2410.09114","locator":"§2 Methodology","version":"Selected original resource https://arxiv.org/html/2410.09114; sha256:b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f; archived source capture; exact capture time not independently established","artifactSha256":"b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f"},{"sourceId":"catastrophic-cyber-benchmark-publication-author-source","url":"https://arxiv.org/html/2410.09114","locator":"§3.2 Elicitation results","version":"Selected original resource https://arxiv.org/html/2410.09114; sha256:b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f; archived source capture; exact capture time not independently established","artifactSha256":"b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f"},{"sourceId":"catastrophic-cyber-benchmark-publication-author-source","url":"https://arxiv.org/html/2410.09114","locator":"§4 Discussion","version":"Selected original resource https://arxiv.org/html/2410.09114; sha256:b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f; archived source capture; exact capture time not independently established","artifactSha256":"b48d6428cc9f577412b3bb2c8bca9f4df649e94e9c2472006f863d8e2ed4046f"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"a5b5f252196be11a1dc821d2411897673757008ec8f62bebac72dd63bab8b59b"}},"sources":[{"id":"catastrophic-cyber-benchmark-publication","title":"Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities","url":"https://arxiv.org/abs/2410.09114","version":"arXiv:2410.09114v2","publicationDate":"2024-10-10","datePrecision":"day","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2410.09114","authors":["Your Name"],"aliases":["https://doi.org/10.48550/arXiv.2410.09114","https://arxiv.org/abs/2410.09114","https://arxiv.org/pdf/2410.09114"]}]}