NUWA 风险评测 · 评测基准
CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning
NUWA 研究解读 · 独立复核待完成
NUWA Research Note
研究问题研究设计与假设来源发现NUWA 解读局限与相反证据研究与决策意义机构与角色原始资源模型辅助整理 · 选读原始全文或官方文章中的相关章节；尚未完成逐章节审计。
研究速读
NUWA 解读：防御推理证据不同于自主攻击成功。
研究问题
模型能否支持核心安全运营防御推理任务？
研究设计与假设
两个组件测恶意软件分析和威胁情报推理，使用原始权重，并有限调整回答格式提示。
来源发现
套件明确测量两类精选人工SOC任务。
部分模型原输出难解析，需分别调整格式提示（§4）。
NUWA 解读
NUWA 解读：防御推理证据不同于自主攻击成功。
局限与相反证据
两类任务不能证明端到端SOC自动化或分析员生产率；格式调整影响纯模型归因。
研究与决策意义
模型建议：展示组件得分、解析处理及提示调整。
精确来源位置
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §3 BenchmarksSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §4 Experimental Setup and ResultsSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §5.4 Additional LimitationsSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §3.1 Malware Analysis Benchmark: Test case structure and metricsSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
来源核查、数据提取、编辑批准与实验复现分别记录。 方法说明 →
机构与角色
作者 (23)
Lauren Deason · Adam Bali · Ciprian Bejean · Diana Bolocan · James Crnkovich · Ioana Croitoru · Krishna Durai · Chase Midler · Calin Miron · David Molnar · Brad Moon · Bruno Ostarcevic · Alberto Peltea · Matt Rosenberg · Catalin Sandu · Arthur Saputkin · Sagar Shah · Daniel Stan · Ernest Szocs · Shengye Wan · Spencer Whitman · Sven Krasser · Joshua Saxe
评测协议
引用与复用
引用研究发现时请引用原始出版物；引用 NUWA 的整理与解释时，单独引用本站记录及对应快照。
原始出版物 · BibTeX
@misc{cybersoceval,
title = {CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning},
url = {https://arxiv.org/html/2509.20166v1},
author = {Lauren Deason and Adam Bali and Ciprian Bejean and Diana Bolocan and James Crnkovich and Ioana Croitoru and Krishna Durai and Chase Midler and Calin Miron and David Molnar and Brad Moon and Bruno Ostarcevic and Alberto Peltea and Matt Rosenberg and Catalin Sandu and Arthur Saputkin and Sagar Shah and Daniel Stan and Ernest Szocs and Shengye Wan and Spencer Whitman and Sven Krasser and Joshua Saxe},
version = {arXiv:2509.20166v1},
note = {arXiv:2509.20166v1}
}
NUWA 记录 · BibTeX
@misc{cybersoceval-nuwa,
title = {CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning — NUWA research record},
url = {https://whitzard.tech/nuwa/whitzard-index/work/cybersoceval/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--cybersoceval.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--cybersoceval.json.}
}
证据快照 2026-10-29.1 · 索引更新 · RSS
参考文献
- [1] Lauren Deason, Adam Bali 等. CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning. v1 ; §3 Benchmarks; §4 Experimental Setup and Results; §5.4 Additional Limitations
- [2] CyberSOCEval Benchmark Data ; Original project README: joint project between CrowdStrike and Meta