{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"cybersoceval","title":"CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning","aliases":["CyberSOCEval"],"authors":["Lauren Deason","Adam Bali","Ciprian Bejean","Diana Bolocan","James Crnkovich","Ioana Croitoru","Krishna Durai","Chase Midler","Calin Miron","David Molnar","Brad Moon","Bruno Ostarcevic","Alberto Peltea","Matt Rosenberg","Catalin Sandu","Arthur Saputkin","Sagar Shah","Daniel Stan","Ernest Szocs","Shengye Wan","Spencer Whitman","Sven Krasser","Joshua Saxe"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"cybersoceval-publication","attributionSourceId":"cybersoceval-publication","originalUrl":"https://arxiv.org/html/2509.20166v1","publicationDate":null,"updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.","zh":"主维度依据原协议的任务结果确定。主题关联见研究档案，不推断尚未审阅的问题级关联。"},"protocolIds":["cybersoceval"],"contributions":[{"organizationId":"meta","role":"research","sourceId":"cybersoc-project-source","locator":"Original project README: joint project between CrowdStrike and Meta"},{"organizationId":"crowdstrike","role":"research","sourceId":"cybersoc-project-source","locator":"Original project README: joint project between CrowdStrike and Meta"}],"summary":{"en":"Evaluates malware analysis and threat-intelligence reasoning as defensive cybersecurity tasks.","zh":"评测恶意软件分析和威胁情报推理等防御性网络安全任务。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"cybersoceval","title":"CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning","en":{"question":"Can models support core defensive security-operations reasoning tasks?","design":"Two components test malware analysis and threat-intelligence reasoning; models use original weights with limited response-format prompt adjustments.","findings":["The suite explicitly measures two selected manual SOC task families.","Per-model formatting adjustments were needed when responses were otherwise unparseable (§4)."],"interpretation":"NUWA interpretation: defensive reasoning evidence is distinct from autonomous offensive success.","limits":"Two task families do not establish end-to-end SOC automation or analyst productivity; prompt-format adjustments complicate pure model attribution.","decisionRelevance":"Model recommendation: keep component scores, parsing treatment and prompt adjustments visible.","takeaway":"NUWA interpretation: defensive reasoning evidence is distinct from autonomous offensive success.","connections":[],"scoring":"Malware-analysis accuracy requires selecting every correct option and no incorrect options; threat-intelligence task scoring is not fully audited here.","conditions":"JSON detonation logs with multiple-response questions; model-specific formatting adjustments.","statistics":"Per-model formatting adjustments were needed when responses were otherwise unparseable (§4).","usage":"Model recommendation: keep component scores, parsing treatment and prompt adjustments visible."},"zh":{"question":"模型能否支持核心安全运营防御推理任务？","design":"两个组件测恶意软件分析和威胁情报推理，使用原始权重，并有限调整回答格式提示。","findings":["套件明确测量两类精选人工SOC任务。","部分模型原输出难解析，需分别调整格式提示（§4）。"],"interpretation":"NUWA 解读：防御推理证据不同于自主攻击成功。","limits":"两类任务不能证明端到端SOC自动化或分析员生产率；格式调整影响纯模型归因。","decisionRelevance":"模型建议：展示组件得分、解析处理及提示调整。","takeaway":"NUWA 解读：防御推理证据不同于自主攻击成功。","connections":[],"scoring":"恶意软件分析须全选正确且不选错误才计正确；威胁情报任务判分此次未完整审计。","conditions":"JSON执行日志及多选题，部分模型分别调整输出格式。","statistics":"部分模型原输出难解析，需分别调整格式提示（§4）。","usage":"模型建议：展示组件得分、解析处理及提示调整。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"cybersoceval-publication","url":"https://arxiv.org/html/2509.20166v1","locator":"§3 Benchmarks","version":"Selected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established","artifactSha256":"854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad"},{"sourceId":"cybersoceval-publication","url":"https://arxiv.org/html/2509.20166v1","locator":"§4 Experimental Setup and Results","version":"Selected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established","artifactSha256":"854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad"},{"sourceId":"cybersoceval-publication","url":"https://arxiv.org/html/2509.20166v1","locator":"§5.4 Additional Limitations","version":"Selected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established","artifactSha256":"854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad"},{"sourceId":"cybersoceval-publication","url":"https://arxiv.org/html/2509.20166v1","locator":"§3.1 Malware Analysis Benchmark: Test case structure and metrics","version":"Selected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established","artifactSha256":"854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"31d2e97190dda70b92a48106910f808013110f2b79ac7df4e4ee769cf2ab593c"}},"sources":[{"id":"cybersoceval-publication","title":"CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning","url":"https://arxiv.org/html/2509.20166v1","version":"arXiv:2509.20166v1","publicationDate":null,"datePrecision":"unknown","type":"paper","verifiedOn":"2026-10-25","documentId":"arxiv:2509.20166","authors":["Lauren Deason*†","Adam Bali†","Ciprian Bejean‡","Diana Bolocan*‡","James Crnkovich*†","Ioana Croitoru*‡","Krishna Durai†","Chase Midler‡","Calin Miron‡","David Molnar*†","Brad Moon‡","Bruno Ostarcevic‡","Alberto Peltea‡","Matt Rosenberg‡","Catalin Sandu‡","Arthur Saputkin†","Sagar Shah*†","Daniel Stan‡","Ernest Szocs‡","Shengye Wan†","Spencer Whitman†","Sven Krasser‡","Joshua Saxe*†"],"aliases":["https://doi.org/10.48550/arXiv.2509.20166","https://arxiv.org/abs/2509.20166","https://arxiv.org/pdf/2509.20166"]}]}