NUWA 来源档案
CAISI Evaluation of DeepSeek V4 Pro
对 CAISI · DeepSeek V4 Pro 的逐来源提取与核查，发现和数值均保留原始归属。
范围与归属
- 数据报告方
- NIST / CAISI
- 实际评测方
- 各观测的执行方未完整确定
- 抓取版本
- 2026-10-26
- 来源身份
- ca064682a53ac9b5
来源转录不等于实验复现。未确定的日期、执行方及配置保留为未知；原文分组不作为经核验排名。
来源发现
以下陈述整理自原始来源，未作为 NUWA 独立实验结论。
CAISI 在其评测套件中将 V4 列为已评测中国模型中能力最高者，并认为它落后于美国前沿趋势；这一结论仅适用于该套件。
Introduction; Key Findings; Figure 1结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
CAISI 将预先确定的评测套件与开发者自行选择、报告的基准比较，并指出两者的结果不同。
Comparison of DeepSeek and CAISI Evaluations; Figure 3结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
成本比较未纳入采用连续评分的 PortBench，也未纳入 GPT-5.4 mini 运行出现技术问题的 ARC-AGI-2。
Cost section结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
CAISI 报告使用云端 H200/B200 与开发者推荐的推理设置，并通过复现 GPQA-Diamond 结果检查推理配置；未给出具体复现分数。
Model Serving and Inference结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
本次比较覆盖五个领域的九项基准。CTF-Archive-Diamond 来源池包含 285 项挑战，该数量不能直接当作每个模型分数的分母。
Capability Results; Benchmarks结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
图 1 使用更广泛的模型与基准数据拟合能力，附录采用 1PL IRT，并在领域内及领域间等权加权；图中误差线为 95% 置信区间。
Figure 1 caption; Appendix结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
CAISI 根据拟合的综合能力趋势，将能力差距描述为约八个月；这是来源的统计解释，不是直接测得的现实风险。
Introduction; Figure 1; Capability Lag Measurement结论范围限于原始来源的任务与实验条件；原文定位保留供核查。
提取范围
以下条目说明来源各部分的处理状态。个别观测已核验，不代表整份文档的全部结论和结果均已验证。
查看 13 项覆盖记录
| 章节 | 处理状态 | 定位与范围 |
|---|---|---|
| Title and introduction | extracted | Page title; introduction; footerPublication 2026-05-01; revision 2026-05-02; DeepSeek evaluation April 2026, exact dates unknown. |
| Figure 1 aggregate capability trend | qualitative | Figure 1 and captionFigure visually inspected. No per-point numerical Elo/date labels; no plot-coordinate estimates. Caption records 16 benchmarks, 35 models, 95% CIs; 200 Elo points corresponds to 3x solve odds. About 8-month lag is CAISI fitted-trend interpretation, not direct risk. |
| Key Findings | qualitative | Key FindingsCAISI comparative claims retained with bounded suite and third-party attribution. |
| Figure 2 full benchmark table | extracted | Capability Results, HTML table 0All 36 benchmark cells (35 numeric, one dash), plus four IRT Elo estimates and printed 95% CI widths. DeepSeek CTF is IRT-imputed. |
| Benchmark definitions | extracted | BenchmarksNine benchmarks/five domains; CTF pool 285 tasks is benchmark metadata, not inferred per-model denominator. No operational task details or biology-specific content extracted. |
| Capability Lag Measurement | qualitative | Capability Lag MeasurementAggregate 1PL IRT; fitted frontier comparison rather than direct elapsed-performance measurement. |
| Model Serving and Inference | extracted | Model Serving and InferenceDeepSeek H200/B200 hosting; developer-recommended inference settings, but exact context length/sampling values absent. ReAct agent; 1M weighted tokens for CTF/PortBench, 500k for SWE. No trial count inferred. |
| Reused developer-selected suite | reused | Figure 3(a)All 26 printed numeric labels. Codeforces Gemini and SWE Gemini not printed; no values estimated. These differ in origin/configuration from CAISI Figure 2. |
| CAISI suite comparison | extracted | Figure 3(b)All eight printed numeric labels; no GPT-5.4 PortBench value. Precision and comparator identity retained separately from Figure 2. |
| Task-cost comparison | extracted | Cost section; Figure 4Seven printed ratios and fourteen printed dollar amounts; matched-correct subset, context only. Two excluded benchmarks and selection method retained. |
| Developer token price assumptions | reused | HTML table 1All six historical developer-reported prices; not independently verified live prices. |
| 1PL IRT methodology | qualitative | AppendixEqual weighting within domain and across five domains; controlled budgets/scaffolds. More detailed writeup promised, no item-level input matrix supplied. |
| Earlier DeepSeek AI Models report | not-relevant | Linked report; Section 3 / Appendix A2 referencesPrior report is a methodological dependency, not evidence for new V4 score gaps. Not ingested wholesale; no additional model scores imported. |
来源待核实事项
DeepSeek 的 CTF-Archive-Diamond 结果由部分样本通过 IRT 插补，不能当作直接观测的完整任务成功率。
图 2 和图 3 对同一结果采用不同显示精度，不代表两次独立实验。
版本指纹
https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro
f2e80121c535923dd74a4c7f7ef069cc72574cbaa7649dc1a223713378c7f8aahttps://www.nist.gov/sites/default/files/styles/1400_x_1400_limit/public/images/2026/05/01/1-Overall-AI-Capability.png.webp?itok=PIASZIV4
17abd969ca69e67c0d049b5ef223c2efcc16736cc6f0baf566087eb92756d2cbhttps://www.nist.gov/sites/default/files/styles/1400_x_1400_limit/public/images/2026/05/01/combined_benchmark_plot.png.webp?itok=La4UVET4
e7388444b6decdc27c5bea81c5cfef9ce087d95f4f5b73e16ec228e275898d15https://www.nist.gov/sites/default/files/styles/1400_x_1400_limit/public/images/2026/05/01/4-Cost-Comparison.png.webp?itok=xW2j7WDH
5884fcda96b812c52fa647290cce24d6a58ad2c007c09b01497c259589db448b参考文献
- [1] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026