CyberSOCEval

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

Can models support core defensive security-operations reasoning tasks?

Task population

Two components test malware analysis and threat-intelligence reasoning; models use original weights with limited response-format prompt adjustments.

Scoring and failure modes

Malware-analysis accuracy requires selecting every correct option and no incorrect options; threat-intelligence task scoring is not fully audited here.

System conditions

JSON detonation logs with multiple-response questions; model-specific formatting adjustments.

Statistical meaning

Source-reported result statements; this does not establish trial counts, denominators or an estimator: Per-model formatting adjustments were needed when responses were otherwise unparseable (§4).

Validity and interpretation

Two task families do not establish end-to-end SOC automation or analyst productivity; prompt-format adjustments complicate pure model attribution.

Appropriate research use

Model recommendation: keep component scores, parsing treatment and prompt adjustments visible.

• CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §3 BenchmarksSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
• CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §4 Experimental Setup and ResultsSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
• CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §5.4 Additional LimitationsSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established
• CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning · §3.1 Malware Analysis Benchmark: Test case structure and metricsSelected original resource https://arxiv.org/html/2509.20166v1; sha256:854341e9f3b6343a27d71f3c8e6c91ad3f712587d0282af67a52f65193a81aad; archived source capture; exact capture time not independently established