NUWA Evidence record
DeepSeek V3.1
Cybench · Evidence record
Source snapshot, chart and review scope →
Original source charts and materials
Recorded measurements
- Reported benchmark performance
- 40%
Source denominator: 40 (unit not disclosed)
CAISI ReAct-style agent; Inspect · 500,000 weighted tokens per agentic task (Appendix A2)
Assessment provenance
Evaluation of DeepSeek AI Models → · External report
Actual evaluation: NIST / CAISI · Model record →
Method and scope
Task completion on 40 professional CTF problems.
- Evidence type
- Benchmark
- Assessment subject
- AI agent solving CTF tasks
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.
Source finding
Reported benchmark performance: 40 percent. Conditions are retained from the source.
Limitations
Competition tasks and supplied guidance differ from operational attacks.
Report labels these as accuracy / percentage of tasks solved. Appendix A3 describes mixed-effects logistic regression estimates and model-based standard errors; do not reconstruct empirical successes or treat this as raw pass@1 counts. No exact API snapshot identifier or per-model trial count disclosed here; benchmark task count is not number of repeated trials. Partial benchmark/harness-specific performance, not measured autonomous real-world risk. Whole-task sandbox benchmark success; not a measurement of all cyber capability or actual harm. CAISI modified these benchmarks and agent setup.
Related research questions
Cybersecurity / Cyber task execution →
The source directly measures the stated construct under its protocol.
References
- [1] Andy K. Zhang, Neil Perry, et al.. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. 2025-04-12. v4 [arXiv]; Author affiliations on the original paper title page
- [2] Evaluation of DeepSeek AI Models. 2025-09-30. 30 September 2025 ; p. 24, Table 4.1, row Cybench, column DeepSeek V3.1