Source record

Evaluation of DeepSeek AI Models

Source-specific extraction of CAISI · DeepSeek models. Findings and measurements retain their original attribution.

Original source and provenance ↓ · Assessment record →

Scope and attribution

Reported by: NIST / CAISI
Actual evaluator: Not established for every observation
Captured version: 2026-10-26
Source identity: 23f49288b6f55815

Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.

Source findings

The following statements summarize the source. They are not independent NUWA experimental findings.

CAISI conducted evaluations in September 2025; the named evaluator is CAISI at NIST, while DeepSeek/OpenAI/Anthropic are model developers.

pp. 2, 4-5

Scope is limited to the source conditions.

DeepSeek weights were downloaded from Hugging Face and self-hosted on CAISI cloud servers. Reference models were queried using cloud APIs; DeepSeek API prices were used for cost comparison.

p. 5

Exact reference API providers/model snapshot identifiers not enumerated.

Agentic cyber results are completion rates in bounded sandbox tasks, while hijacking measures willingness to attempt an injected action and jailbreaking measures response compliance/detail. These are distinct constructs.

pp. 24-30, 46, 50

No construct establishes general real-world cyber risk or dangerous capability by itself.

CAISI used a custom CVE-Bench subset, modified Cybench to use Inspect and its own agent, and converted pwn.college tasks into a benchmark; original public benchmark results are not interchangeable.

pp. 25, 27, 29

Scope is limited to the source conditions.

CAISI reports weaker DeepSeek agentic benchmark performance than leading reference models, with improvements from R1 to V3.1; the finding is bounded to the evaluated models, settings and tasks.

pp. 6, 8-10, 24, 31

Scope is limited to the source conditions.

The report describes GLMM/mixed-effects logistic regression and delta-method errors. It does not call its scores IRT estimates. Do not rename or present inferred empirical pass counts.

Appendix A3, pp. 64-65

The text describes both empirical averaging and model-based estimation, so exact mapping of every table point to an estimator is not fully explicit.

HealthBench methods use a weighted physician-rubric score and GPT-4.1 judge, even though the summary table broadly calls its values percentage of tasks solved.

pp. 34, 38

Keep original label and scoring qualification together.

Hijacking study uses fixed public and non-public injections, rather than adaptive red teaming of evaluated models; cases that never encounter the injection are excluded, and attempted harmful actions count even if unsuccessful.

p. 46

Here no exposure means failure to encounter malicious text; excluded denominator varies by model/task.

For non-biological harmful requests, compliance/detail grading measures response behavior; detail does not measure correctness or counterfactual utility. Three model graders were combined by mode, with validation agreement of 96% for compliance and 84% for detail.

p. 50

No detailed harmful queries or operational jailbreak text copied.

The narrative-alignment benchmark uses U.S. Department of State expert-selected flags and LLM scoring; results depend on the narrative set and may change as the benchmark evolves.

pp. 53-55

Scope is limited to the source conditions.

Adoption measures are partial platform proxies. Derivative uploads refer to fine-tuned or otherwise modified downstream models and do not establish the performance/security properties of the original or every derivative.

pp. 23, 56-60

Scope is limited to the source conditions.

Findings are preliminary, partial and specific to model versions and time; report disclaims certification and endorsement.

p. 61

Scope is limited to the source conditions.

Extraction coverage

Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.

View 105 coverage entries

Section	Treatment	Location and scope
Executive Summary	qualitative	p. 2Non-CBRN summary claims retained; cross-domain misuse percentages omitted because harmful biology is included.
Methodology	extracted	pp. 4-5Independent CAISI evaluation; self-hosted DeepSeek vs cloud APIs for references; model identity and deployment caveats retained.
Overview of Findings	qualitative	p. 6Overview maps to detailed sections; no additional independent experiment.
Performance Evaluations Overview	extracted	pp. 7-13All Table 3.1 cells retained; rounded narrative duplicates not duplicated as independent evidence.
Cost Efficiency Analysis Overview	extracted	pp. 14-15Aggregate and price/context caveats retained; CI figure has no numeric endpoints.
Security Evaluations Overview	qualitative	pp. 16-20Narrow assessment; does not test sleeper triggers; self-hosted deployment caveat.
Agent Hijacking	extracted	pp. 16-17Nine printed prose percentages retained; no bar height estimation.
Jailbreaking	qualitative	pp. 18-20CBRN limited to metadata/high-level caveats; exact non-biological prose outputs retained separately.
CCP Censorship Overview	qualitative	p. 21Source interpretation attributed; exact detailed Table 7.5 retained instead of rounded summary numbers.
Model Adoption Overview	extracted	pp. 22-23Printed platform/time-window statistics retained; detailed section supplies repeated values.
Performance Evaluations	extracted	pp. 24-41All four numeric tables and task-set methods retained.
Cyber	extracted	pp. 24-30Whole-task agentic success metrics; no task/exploit instructions copied.
CVE-Bench	extracted	pp. 25-26Custom 15-task set: seven public and eight private; pass/fail objectives; not original public benchmark configuration.
Cybench	extracted	pp. 27-2840 challenges; CAISI Inspect/agent architecture; missing original competition files restored.
CTF-Archive	extracted	pp. 29-30577 challenges from 79 competitions (2011-2025); CAISI conversion; category chart figure-only.
Software Development	extracted	pp. 31-34Table 4.8 all cells retained.
SWE-bench Verified	extracted	pp. 31-32489 problems from 12 repositories; excluded 11 original problems as Anthropic did; success requires all tests.
Breakpoint	extracted	pp. 33-34200-task subset; code-corruption framework; success requires all tests.
Science and Knowledge	extracted	pp. 34-39Table 4.13 all cells retained, source internal conflicts quarantined.
MMLU-Pro	extracted	p. 35Same 1,000 randomly sampled science-discipline questions for all models; not full 12,032-question benchmark.
MMMLU	extracted	p. 36Same 1,400 questions: 100 per 14 languages; all disciplines.
GPQA	extracted	p. 37GPQA-Diamond 198 questions; general knowledge score only; no biological task details.
HealthBench	extracted	p. 385,000 conversations; 48,562 rubric criteria; GPT-4.1 grader; rubric score cannot be treated as binary solves.
Humanity’s Last Exam	extracted	p. 392,158 text-only questions, not full 2,500 question/multimodal dataset.
Mathematical Reasoning	extracted	pp. 40-41Table 4.19 all cells retained; o4-mini equivalence grader.
Recent mathematics competitions	extracted	pp. 40-41SMT 58; OTIS-AIME 30; PUMaC 55 text-only tasks.
Cost Efficiency Analysis	extracted	pp. 42-44All Table 5.3 cells; curves not digitized; geometric expense ratios over overlapping achievable performance range.
Security Evaluations	qualitative	pp. 45-52Hijacking vs refusal/detail scoring separated; not complete security assessment.
Agent Hijacking	extracted	pp. 45-4830 benign tasks x five samples x two fixed injections, three malicious task classes; attempted actions conditional on injection exposure; default AgentDojo enhanced by CAISI.
Jailbreaking	qualitative	pp. 48-52Harmful biology/violence dataset: methodological metadata only; no dangerous capability comparisons or task content. Non-biological 30 hacking +50 scam queries; 17 public attacks, separate selection and evaluation sets; three graders. No operational prompts copied.
Censorship Evaluations	extracted	pp. 53-55190 free responses; narrative-flag rubric and LLM judge; all Table 7.5 cells. Tables 7.1-7.3 illustrative methodology only.
Model Adoption	extracted	pp. 56-60All exact printed prose statistics retained; charts not digitized.
Model Downloads	extracted	pp. 56-57Hugging Face/ModelScope publicly archived counts; downloads do not equal users.
Model Usage via API	extracted	pp. 57-59OpenRouter family/variant aggregation and limited platform coverage retained.
Derivative Model Uploads	extracted	pp. 59-60Exact prose relative bound retained; plot points figure-only; modification does not establish original model risk/performance.
Disclaimer	qualitative	p. 61Preliminary, partial, time/version-specific; benchmark contamination and generalization caveats; no certification or endorsement.
Agent Design	extracted	pp. 62-63CAISI ReAct; tool formats optimized model-by-model; same benchmark environment and tools across models.
Agent Resource Budget	extracted	pp. 63-64500,000 weighted tokens: output 100%, unique input 23%, total input 2%; not raw output token ceiling.
Uncertainty Estimation	extracted	pp. 64-65Mixed-effects logistic regression, model fixed effects/task random effects; delta-method SE. Not labelled IRT; no raw per-cell counts inferred.
Model Parameters	extracted	p. 65Model-specific agentic/nonagentic modes captured; DeepSeek temperature 0.6/top_p 0.95; Opus agentic temperature 0.7.
Model Token Prices	extracted	p. 66All six Table A5.1 values; historical authoring-time quote.
Cost Ratio Calculation	extracted	pp. 66-67Expense curves estimated from transcripts; truncated at alternate budget trigger; geometric mean over common achievable performance interval.
Selection of Benchmarks	qualitative	pp. 67-68Lower-contamination/real-world relevance rationale; multimodal inputs excluded; metadata only for CBRN.
Comparison to Self-Reported Benchmark Scores	reused	pp. 68-69All nonblank cells retained; Self columns explicitly reused developer results; no origin revalidation; blank cells not zero.
Table 3.1: Capability overview	extracted	p. 7All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.1: Cyber performance	extracted	p. 24All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.8: Software performance	extracted	p. 31All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.13: Science and knowledge performance	extracted	p. 34All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.19: Mathematics performance	extracted	p. 40All printed nonblank numeric cells retained; duplicated results identified as such.
Table 5.3: Performance and cost ratios	extracted	p. 44All printed nonblank numeric cells retained; duplicated results identified as such.
Table 7.5: Narrative alignment	extracted	p. 55All printed nonblank numeric cells retained; duplicated results identified as such.
Table A5.1: Token prices	extracted	p. 66All printed nonblank numeric cells retained; duplicated results identified as such.
Table A8.1: CAISI/self comparison	reused	p. 69All printed nonblank numeric cells retained; duplicated results identified as such.
Table 7.1: Example bilingual questions	qualitative	p. 53Illustrative method, not a measured model score; no synthetic example promoted to an observation.
Table 7.2: Example narrative flags	qualitative	p. 53Illustrative method, not a measured model score; no synthetic example promoted to an observation.
Table 7.3: Worked narrative alignment illustration	qualitative	p. 54Illustrative method, not a measured model score; no synthetic example promoted to an observation.
Figure 3.2	figure-only	p. 8No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.3	figure-only	p. 10No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.4	figure-only	p. 11No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.5	figure-only	p. 13No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.6	figure-only	p. 14No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.7	figure-only	p. 17No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.8	qualitative	p. 19CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 3.9	qualitative	p. 19CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 3.10	figure-only	p. 19No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.11	figure-only	p. 20No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.12	figure-only	p. 21No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.13	figure-only	p. 22No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.2	figure-only	p. 24No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.3	figure-only	p. 26No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.4	figure-only	p. 26No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.5	figure-only	p. 28No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.6	figure-only	p. 28No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.7	figure-only	p. 30No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.9	figure-only	p. 32No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.10	figure-only	p. 32No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.11	figure-only	p. 33No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.12	figure-only	p. 34No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.14	figure-only	p. 35No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.15	figure-only	p. 36No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.16	figure-only	p. 37No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.17	figure-only	p. 38No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.18	figure-only	p. 39No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.20	figure-only	p. 40No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.21	figure-only	p. 41No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.22	figure-only	p. 41No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 5.1	figure-only	p. 42No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 5.2	figure-only	p. 43No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.1	figure-only	p. 45No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.2	figure-only	p. 47No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.3	figure-only	p. 48No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.4	figure-only	p. 49No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.5	figure-only	p. 49No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.6	qualitative	p. 51CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 6.7	qualitative	p. 51CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 6.8	figure-only	p. 52No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.9	figure-only	p. 52No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 7.4	figure-only	p. 55No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.1	figure-only	p. 56No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.2	figure-only	p. 57No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.3	figure-only	p. 58No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.4	figure-only	p. 59No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.5	figure-only	p. 59No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.6	figure-only	p. 60No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure A1.1	figure-only	p. 62No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.

Unresolved source issues

本报告使用 CVE-Bench、Cybench 和 CTF-Archive，未评测 CyberGym。Cybench 的实际方法位于 §4.1.2。

报告同时描述经验平均与混合效应估计；任务总数不能直接用来反推表中分数的成功次数。

Version fingerprints

document

https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf

0577019513c6636753deb7818f0af7e4534a68217bb1c45b9baaed0759b01ecc

document

https://www.nist.gov/news-events/news/2025/09/caisi-evaluation-deepseek-ai-models-finds-shortcomings-and-risks

615269ccc21748ab4e6d24972adb4baa2a666c1a858d8fac5b4f7129ac48f58d

renderer

https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf

00fc244f57cc82601b2f7ff65ac22a3f7d6c9d63c2e362643655b7c026da6f36

renderer

https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf

fd4eb9d01d18939a7cde29305bd6060efabde5edda633ab8a09336dae4957505

renderer

https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf

f296506e26546871af08020316ab7abcd2cf01bab25c1d7e1c2dd502a139851a

data

https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf

4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af

References

[1] Evaluation of DeepSeek AI Models. 2025-09-30. 30 September 2025