NUWA Evidence record
DeepSeek v4 Pro
Cybench · Evidence record
Source snapshot, chart and review scope →
Original source charts and materials
Recorded measurements
- pass@1
- 79.5%95% CI bars displayed; numeric endpoints not disclosed
Source denominator: 39 (unit not disclosed)
Inspect AI · 3 submission attempts; unlimited wall-clock time
Assessment provenance
Evaluating DeepSeek v4 Pro for Frontier Risks → · External report
Actual evaluation: Neo Research · Model record →
Method and scope
Task completion on 40 professional CTF problems.
- Evidence type
- Benchmark
- Assessment subject
- AI agent solving CTF tasks
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.
Source finding
pass@1: 79.5 percent. Conditions are retained from the source.
Limitations
Competition tasks and supplied guidance differ from operational attacks.
Pass@1 naming is retained despite a stated three-submission budget.
Related research questions
Cybersecurity / Cyber task execution →
The source directly measures the stated construct under its protocol.
References
- [1] Andy K. Zhang, Neil Perry, et al.. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. 2025-04-12. v4 [arXiv]; Author affiliations on the original paper title page
- [2] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 8; Cybench unguided; pass@1