NUWA Risk Evaluation · Technical report
CAISI Evaluation of DeepSeek V4 Pro
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: the lag is a model-based historical summary, not uniform underperformance on every task.
Research question
How does an independently selected evaluation suite change the picture of DeepSeek V4 Pro?
Design and assumptions
Nine benchmarks span five domains, with two held-out datasets; a 1PL IRT fit weights benchmarks within domains and domains equally.
Source findings
The report estimates an approximately eight-month aggregate capability lag in its chosen suite.
It reproduces GPQA-Diamond to check inference/configuration and notes prompt/scaffold/budget differences for SWE-bench.
NUWA interpretation
NUWA interpretation: the lag is a model-based historical summary, not uniform underperformance on every task.
Limits and contrary evidence
CTF results include IRT imputation and PortBench continuous scoring; cost analysis applies only to seven eligible benchmarks and shared solved tasks.
Research and decision relevance
Model recommendation: retain benchmark-level evidence, IRT assumptions and exclusions with any aggregate claim.
Precise source locations
- CAISI Evaluation of DeepSeek V4 Pro · Capability ResultsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CAISI Evaluation of DeepSeek V4 Pro · BenchmarksSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CAISI Evaluation of DeepSeek V4 Pro · Capability Lag MeasurementSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CAISI Evaluation of DeepSeek V4 Pro · Comparison of DeepSeek and CAISI EvaluationsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CAISI Evaluation of DeepSeek V4 Pro · DeepSeek V4 Costs Less Than Other Models Of Similar CapabilitySelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CAISI Evaluation of DeepSeek V4 Pro · AppendixSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{caisi-deepseek-v4-report,
title = {CAISI Evaluation of DeepSeek V4 Pro},
url = {https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro},
year = {2026},
date = {2026-05-01},
version = {1 May 2026},
note = {1 May 2026}
}
NUWA record · BibTeX
@misc{caisi-deepseek-v4-report-nuwa,
title = {CAISI Evaluation of DeepSeek V4 Pro — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/caisi-deepseek-v4-report/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--caisi-deepseek-v4-report.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--caisi-deepseek-v4-report.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026 ; Report masthead and stated evaluation methodology; Capability Results; Benchmarks