NUWA EXTERNAL ASSESSMENT · NIST / CAISI
Evaluation of DeepSeek AI Models
Reported differences depend on task, scaffold and scoring; institutional judgments are retained as source judgments rather than NUWA findings.
Published 2025-09-30
Sources ↓Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.
Models covered
NUWA Research Note
Research question
How do selected DeepSeek releases perform under CAISI’s capability and security protocols?
Method
The September 2025 report covers public and held-out tasks, inference configuration, costs, hijacking and jailbreak evaluations.
Source-reported findings
Reported differences depend on task, scaffold and scoring; institutional judgments are retained as source judgments rather than NUWA findings.
Limitations
Controlled tasks, elicitation, benchmark familiarity and configuration differences constrain generalization to deployment.
Cybench
Task completion on 40 professional CTF problems.
Competition tasks and supplied guidance differ from operational attacks.
External evaluation: NIST / CAISI
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
CTF-Archive
CTF-Archive in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
SWE-Bench Verified
SWE-Bench Verified in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Breakpoint
Breakpoint in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
MMLU-Pro
MMLU-Pro in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
MMMLU
MMMLU in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
GPQA-Diamond
GPQA-Diamond in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
HealthBench
HealthBench in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Humanity’s Last Exam
Humanity’s Last Exam in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
SMT 2025
SMT 2025 in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
OTIS-AIME 2025
OTIS-AIME 2025 in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
PUMaC 2024
PUMaC 2024 in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Agent hijacking
Targeted attack success and user-task utility in a tool-use environment.
Attack success must be read alongside benign utility and the tested attack setup.
External evaluation: NIST / CAISI
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Jailbreak robustness
Jailbreak robustness in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Censorship and narrative evaluations
Censorship and narrative evaluations in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Report coverage
| Section | Source location | Treatment |
|---|---|---|
| Cybench | §4.1.1 | Protocol variant Protocol and result provenance are retained at this section. |
| CyberGym | §4.1.2 | Protocol variant Protocol and result provenance are retained at this section. |
| CTF-Archive | §4.1.3 | Contextual evidence Protocol and result provenance are retained at this section. |
| SWE-Bench Verified | §4.2.1 | Contextual evidence Protocol and result provenance are retained at this section. |
| Breakpoint | §4.2.2 | Contextual evidence Protocol and result provenance are retained at this section. |
| MMLU-Pro | §4.3.1 | Contextual evidence Protocol and result provenance are retained at this section. |
| MMMLU | §4.3.2 | Contextual evidence Protocol and result provenance are retained at this section. |
| GPQA-Diamond | §4.3.3 | Contextual evidence Protocol and result provenance are retained at this section. |
| HealthBench | §4.3.4 | Contextual evidence Protocol and result provenance are retained at this section. |
| Humanity’s Last Exam | §4.3.5 | Contextual evidence Protocol and result provenance are retained at this section. |
| SMT 2025 | §4.4 | Contextual evidence Protocol and result provenance are retained at this section. |
| OTIS-AIME 2025 | §4.4 | Contextual evidence Protocol and result provenance are retained at this section. |
| PUMaC 2024 | §4.4 | Contextual evidence Protocol and result provenance are retained at this section. |
| Agent hijacking | §5.1 | Protocol variant Protocol and result provenance are retained at this section. |
| Jailbreak robustness | §5.2 | Contextual evidence Protocol and result provenance are retained at this section. |
| Censorship and narrative evaluations | §6 | Contextual evidence Protocol and result provenance are retained at this section. |
| Model adoption | §7 | Contextual evidence Protocol and result provenance are retained at this section. |
| Inference and scoring | Appendices A1–A3 | Contextual evidence Protocol and result provenance are retained at this section. |
References
Numerical results are reported by the cited authors.
- [1] Evaluation of DeepSeek AI Models. 2025-09-30. 30 September 2025 ; p. 24, Table 4.1, row Cybench, column OpenAI GPT-5; p. 24, Table 4.1, row Cybench, column OpenAI gpt-oss; p. 24, Table 4.1, row Cybench, column DeepSeek V3.1