Source record

CAISI Evaluation of DeepSeek V4 Pro

Source-specific extraction of CAISI · DeepSeek V4 Pro. Findings and measurements retain their original attribution.

Original source and provenance ↓ · Assessment record →

Scope and attribution

Reported by: NIST / CAISI
Actual evaluator: Not established for every observation
Captured version: 2026-10-26
Source identity: ca064682a53ac9b5

Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.

Source findings

The following statements summarize the source. They are not independent NUWA experimental findings.

CAISI ranks V4 highest among evaluated PRC models and behind its U.S. frontier trend; conclusion is restricted to the evaluated suite.

Introduction; Key Findings; Figure 1

Not a universal model ranking or direct real-world harm claim.

CAISI contrasts its precommitted suite with developer-selected, developer-reported benchmark comparisons.

Comparison of DeepSeek and CAISI Evaluations; Figure 3

Figure 3(a) results are reused; source report not linked in article.

PortBench cost omitted because scoring is continuous; ARC-AGI-2 cost omitted due to GPT-5.4 mini run issues.

Cost section

Scope is limited to the source conditions.

CAISI reports cloud H200/B200 serving and developer-recommended inference settings, plus GPQA-Diamond reproduction as an inference check.

Model Serving and Inference

Exact numeric inference configuration and reproduction score/run details not provided.

CAISI describes nine benchmarks in five domains; CTF-Archive-Diamond has a 285-challenge source pool.

Capability Results; Benchmarks

Pool size does not establish the denominator used by any displayed model result.

Figure 1 caption reports 16 benchmarks across 35 models, 95% intervals, and a 200-point/3x solve-odds scale; Appendix specifies 1PL IRT and equal domain weighting.

Figure 1 caption; Appendix

Figure 1 wider fitting corpus differs from the nine-benchmark current comparison.

CAISI describes the capability frontier gap as about eight months using its fitted aggregate-capability comparison.

Introduction; Figure 1; Capability Lag Measurement

Approximate source interpretation; dates/coordinates not estimated from chart.

Extraction coverage

Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.

View 13 coverage entries

Section	Treatment	Location and scope
Title and introduction	extracted	Page title; introduction; footerPublication 2026-05-01; revision 2026-05-02; DeepSeek evaluation April 2026, exact dates unknown.
Figure 1 aggregate capability trend	qualitative	Figure 1 and captionFigure visually inspected. No per-point numerical Elo/date labels; no plot-coordinate estimates. Caption records 16 benchmarks, 35 models, 95% CIs; 200 Elo points corresponds to 3x solve odds. About 8-month lag is CAISI fitted-trend interpretation, not direct risk.
Key Findings	qualitative	Key FindingsCAISI comparative claims retained with bounded suite and third-party attribution.
Figure 2 full benchmark table	extracted	Capability Results, HTML table 0All 36 benchmark cells (35 numeric, one dash), plus four IRT Elo estimates and printed 95% CI widths. DeepSeek CTF is IRT-imputed.
Benchmark definitions	extracted	BenchmarksNine benchmarks/five domains; CTF pool 285 tasks is benchmark metadata, not inferred per-model denominator. No operational task details or biology-specific content extracted.
Capability Lag Measurement	qualitative	Capability Lag MeasurementAggregate 1PL IRT; fitted frontier comparison rather than direct elapsed-performance measurement.
Model Serving and Inference	extracted	Model Serving and InferenceDeepSeek H200/B200 hosting; developer-recommended inference settings, but exact context length/sampling values absent. ReAct agent; 1M weighted tokens for CTF/PortBench, 500k for SWE. No trial count inferred.
Reused developer-selected suite	reused	Figure 3(a)All 26 printed numeric labels. Codeforces Gemini and SWE Gemini not printed; no values estimated. These differ in origin/configuration from CAISI Figure 2.
CAISI suite comparison	extracted	Figure 3(b)All eight printed numeric labels; no GPT-5.4 PortBench value. Precision and comparator identity retained separately from Figure 2.
Task-cost comparison	extracted	Cost section; Figure 4Seven printed ratios and fourteen printed dollar amounts; matched-correct subset, context only. Two excluded benchmarks and selection method retained.
Developer token price assumptions	reused	HTML table 1All six historical developer-reported prices; not independently verified live prices.
1PL IRT methodology	qualitative	AppendixEqual weighting within domain and across five domains; controlled budgets/scaffolds. More detailed writeup promised, no item-level input matrix supplied.
Earlier DeepSeek AI Models report	not-relevant	Linked report; Section 3 / Appendix A2 referencesPrior report is a methodological dependency, not evidence for new V4 score gaps. Not ingested wholesale; no additional model scores imported.

Unresolved source issues

DeepSeek 的 CTF-Archive-Diamond 结果由部分样本通过 IRT 插补，不能当作直接观测的完整任务成功率。

图 2 和图 3 对同一结果采用不同显示精度，不代表两次独立实验。

Version fingerprints

document

https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro

f2e80121c535923dd74a4c7f7ef069cc72574cbaa7649dc1a223713378c7f8aa

data

https://www.nist.gov/sites/default/files/styles/1400_x_1400_limit/public/images/2026/05/01/1-Overall-AI-Capability.png.webp?itok=PIASZIV4

17abd969ca69e67c0d049b5ef223c2efcc16736cc6f0baf566087eb92756d2cb

data

https://www.nist.gov/sites/default/files/styles/1400_x_1400_limit/public/images/2026/05/01/combined_benchmark_plot.png.webp?itok=La4UVET4

e7388444b6decdc27c5bea81c5cfef9ce087d95f4f5b73e16ec228e275898d15

data

https://www.nist.gov/sites/default/files/styles/1400_x_1400_limit/public/images/2026/05/01/4-Cost-Comparison.png.webp?itok=xW2j7WDH

5884fcda96b812c52fa647290cce24d6a58ad2c007c09b01497c259589db448b

References

[1] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026