EXTERNAL ASSESSMENT · NIST / CAISI

CAISI Evaluation of DeepSeek V4 Pro

The assessment reports benchmark-specific performance and an IRT aggregate; the latter is not a frontier-risk score.

Published 2026-05-01

Source version, extraction coverage and findings →

Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.

Models covered

DeepSeek V4 Pro →GPT-5.5 →GPT-5.4 mini →Claude Opus 4.6 →GPT-5.4 →

NUWA Research Note

Research question

How does DeepSeek V4 Pro perform across CAISI’s selected capability tasks?

Method

Nine task families, model-specific reasoning settings and weighted-token budgets; CTF results for DeepSeek include IRT imputation.

Source-reported findings

The assessment reports benchmark-specific performance and an IRT aggregate; the latter is not a frontier-risk score.

Limitations

Controlled tasks, elicitation, benchmark familiarity and configuration differences constrain generalization to deployment.

capability · Capability Results; Figure 2

CTF-Archive-Diamond

Protocol record →

285 difficult CTF tasks; Inspect ReAct with 1M weighted tokens. DeepSeek result is IRT-imputed from a subset.

CTF tasks are not live operational attacks; subset-imputed values are not directly observed full-set success rates.

External evaluation: NIST / CAISI

CTF-Archive-Diamond

Original evaluator results · source order

Task accuracy

0–100%

GPT-5.5 (xhigh)

GPT-5.5 (xhigh) · 71% · CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately. · Inspect ReAct · 285 difficult CTF tasks; DeepSeek score is imputed from a subset.

71%

GPT-5.4 mini (xhigh)

GPT-5.4 mini (xhigh) · 32% · CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately. · Inspect ReAct · 285 difficult CTF tasks; DeepSeek score is imputed from a subset.

32%

Claude Opus 4.6 (max)

Claude Opus 4.6 (max) · 46% · CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately. · Inspect ReAct · 285 difficult CTF tasks; DeepSeek score is imputed from a subset.

46%

DeepSeek V4 Pro (max)

DeepSeek V4 Pro (max) · ≈32% * · CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately. · Inspect ReAct · 285 difficult CTF tasks; DeepSeek score is imputed from a subset.

≈32% *

Showing 4 of 4 systems

CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately.

CTF tasks are not live operational attacks; subset-imputed values are not directly observed full-set success rates. Report dates and original executor are shown with this assessment.

Evaluation conditions and system configurations

GPT-5.5 (xhigh): 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond; modelVersion: GPT-5.5 (xhigh) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens

GPT-5.4 mini (xhigh): 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond; modelVersion: GPT-5.4 mini (xhigh) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens

Claude Opus 4.6 (max): 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond; modelVersion: Claude Opus 4.6 (max) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens

DeepSeek V4 Pro (max): 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond; modelVersion: DeepSeek V4 Pro (max) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens

CTF-Archive-Diamond · source order
System / configuration	Task accuracy	Publication / verification
GPT-5.5 (xhigh)Inspect ReAct	71%	Published2026-05-01
GPT-5.4 mini (xhigh)Inspect ReAct	32%	Published2026-05-01
Claude Opus 4.6 (max)Inspect ReAct	46%	Published2026-05-01
DeepSeek V4 Pro (max)Inspect ReAct	≈32%IRT-imputed, not directly observed.	Published2026-05-01

capability · Capability Results; Figure 2

SWE-Bench Verified

SWE-Bench Verified in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

capability · Capability Results; Figure 2

PortBench

Protocol record →

CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.

The task set is non-public and continuous scoring differs from binary task success.

External evaluation: NIST / CAISI

PortBench

Original evaluator results · source order

Reported task accuracy

0–100%

GPT-5.5 (xhigh)

GPT-5.5 (xhigh) · 78% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

78%

GPT-5.4 mini (xhigh)

GPT-5.4 mini (xhigh) · 41% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

41%

Claude Opus 4.6 (max)

Claude Opus 4.6 (max) · 60% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

60%

DeepSeek V4 Pro (max)

DeepSeek V4 Pro (max) · 44% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

44%

Showing 4 of 4 systems

CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.

The task set is non-public and continuous scoring differs from binary task success. Report dates and original executor are shown with this assessment.

Evaluation conditions and system configurations

GPT-5.5 (xhigh): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: GPT-5.5 (xhigh) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

GPT-5.4 mini (xhigh): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: GPT-5.4 mini (xhigh) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

Claude Opus 4.6 (max): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: Claude Opus 4.6 (max) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

DeepSeek V4 Pro (max): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: DeepSeek V4 Pro (max) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

PortBench · source order
System / configuration	Reported task accuracy	Publication / verification
GPT-5.5 (xhigh)Inspect ReAct	78%	Published2026-05-01
GPT-5.4 mini (xhigh)Inspect ReAct	41%	Published2026-05-01
Claude Opus 4.6 (max)Inspect ReAct	60%	Published2026-05-01
DeepSeek V4 Pro (max)Inspect ReAct	44%	Published2026-05-01

capability · Capability Results; Figure 2

FrontierScience

FrontierScience in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

capability · Capability Results; Figure 2

GPQA-Diamond

GPQA-Diamond in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

capability · Capability Results; Figure 2

ARC-AGI-2 semi-private

ARC-AGI-2 semi-private in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

capability · Capability Results; Figure 2

OTIS-AIME-2025

OTIS-AIME-2025 in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

capability · Capability Results; Figure 2

PUMaC 2024

PUMaC 2024 in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

capability · Capability Results; Figure 2

SMT 2025

SMT 2025 in the report-defined setting.

Exploratory or report-specific setting; not a reusable benchmark claim.

This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.

Report coverage

Section	Source location	Treatment
CTF-Archive-Diamond	Capability Results; Figure 2	Protocol variant Protocol and result provenance are retained at this section.
SWE-Bench Verified	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
PortBench	Capability Results; Figure 2	Protocol variant Protocol and result provenance are retained at this section.
FrontierScience	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
GPQA-Diamond	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
ARC-AGI-2 semi-private	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
OTIS-AIME-2025	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
PUMaC 2024	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
SMT 2025	Capability Results; Figure 2	Contextual evidence Protocol and result provenance are retained at this section.
IRT aggregation and imputation	Appendix; Figure 2 footnotes	Limitations Protocol and result provenance are retained at this section.
Cost and serving configuration	Model Serving and Inference; cost section	Contextual evidence Protocol and result provenance are retained at this section.

References

Numerical results are reported by the cited authors.

[1] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026 ; Capability Results; Figure 2; *** footnote; Capability Results; Figure 2; Figure 2 HTML table, CTF-Archive-Diamond, OpenAI GPT-5.5 (xhigh)