来源档案

Evaluation of DeepSeek AI Models

对 CAISI · DeepSeek models 的逐来源提取与核查，发现和数值均保留原始归属。

原始来源与出处 ↓ · 评估档案 →

范围与归属

数据报告方: NIST / CAISI
实际评测方: 各观测的执行方未完整确定
抓取版本: 2026-10-26
来源身份: 23f49288b6f55815

来源转录不等于实验复现。未确定的日期、执行方及配置保留为未知；原文分组不作为经核验排名。

来源发现

以下陈述整理自原始来源，未作为 NUWA 独立实验结论。

CAISI 于 2025 年 9 月开展评测。评测机构是 NIST 下属 CAISI，DeepSeek、OpenAI 和 Anthropic 是模型开发者。

pp. 2, 4-5

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

CAISI 从 Hugging Face 下载 DeepSeek 权重并在自有云服务器托管；参考模型通过云端 API 调用，成本比较使用 DeepSeek 公布的 API 价格。

p. 5

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

网络能力结果是受限沙箱任务的完成率；劫持评测关注是否尝试执行注入指令；越狱评测关注回应的服从程度与细节。这些测量对象不同。

pp. 24-30, 46, 50

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

CAISI 使用定制 CVE-Bench 子集，将 Cybench 改造为 Inspect 与自有智能体架构，并把 pwn.college 挑战转换为评测任务；结果不能与原协议直接互换。

pp. 25, 27, 29

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

在已评测模型、设置和任务范围内，CAISI 报告 DeepSeek 智能体基准表现落后于领先参考模型，同时 V3.1 相比早期 R1 有所改善。

pp. 6, 8-10, 24, 31

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

报告采用广义线性混合模型与混合效应逻辑回归，并用 delta 方法估计误差，没有将这些分数称为 IRT 估计。不能据此反推成功次数。

Appendix A3, pp. 64-65

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

HealthBench 使用医生评分细则的加权分数与 GPT-4.1 判别器；汇总表笼统的“任务解决百分比”表述不能替代该评分定义。

pp. 34, 38

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

劫持研究使用固定的公开与非公开注入，而非针对本批模型自适应红队。未遇到注入文本的样本被排除，尝试恶意行动即计入，行动是否成功另作区分。

p. 46

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

非生物类恶意请求评测衡量回应的服从与细节；细节分不衡量正确性或实际效用。报告使用三个判别模型的众数，并以人工标注检查一致性。

p. 50

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

叙事一致性基准使用美国国务院专家选择的标记与模型评分；结果依赖于所选叙事集，未来版本可能不可直接比较。

pp. 53-55

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

采用率数据只是部分平台的代理指标。衍生模型上传量不能证明原模型或全部衍生模型的性能与安全性质。

pp. 23, 56-60

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

报告声明结果是初步、局部且依赖版本与时间的评估，不构成认证或背书。

p. 61

结论范围限于原始来源的任务与实验条件；原文定位保留供核查。

提取范围

以下条目说明来源各部分的处理状态。个别观测已核验，不代表整份文档的全部结论和结果均已验证。

查看 105 项覆盖记录

章节	处理状态	定位与范围
Executive Summary	qualitative	p. 2Non-CBRN summary claims retained; cross-domain misuse percentages omitted because harmful biology is included.
Methodology	extracted	pp. 4-5Independent CAISI evaluation; self-hosted DeepSeek vs cloud APIs for references; model identity and deployment caveats retained.
Overview of Findings	qualitative	p. 6Overview maps to detailed sections; no additional independent experiment.
Performance Evaluations Overview	extracted	pp. 7-13All Table 3.1 cells retained; rounded narrative duplicates not duplicated as independent evidence.
Cost Efficiency Analysis Overview	extracted	pp. 14-15Aggregate and price/context caveats retained; CI figure has no numeric endpoints.
Security Evaluations Overview	qualitative	pp. 16-20Narrow assessment; does not test sleeper triggers; self-hosted deployment caveat.
Agent Hijacking	extracted	pp. 16-17Nine printed prose percentages retained; no bar height estimation.
Jailbreaking	qualitative	pp. 18-20CBRN limited to metadata/high-level caveats; exact non-biological prose outputs retained separately.
CCP Censorship Overview	qualitative	p. 21Source interpretation attributed; exact detailed Table 7.5 retained instead of rounded summary numbers.
Model Adoption Overview	extracted	pp. 22-23Printed platform/time-window statistics retained; detailed section supplies repeated values.
Performance Evaluations	extracted	pp. 24-41All four numeric tables and task-set methods retained.
Cyber	extracted	pp. 24-30Whole-task agentic success metrics; no task/exploit instructions copied.
CVE-Bench	extracted	pp. 25-26Custom 15-task set: seven public and eight private; pass/fail objectives; not original public benchmark configuration.
Cybench	extracted	pp. 27-2840 challenges; CAISI Inspect/agent architecture; missing original competition files restored.
CTF-Archive	extracted	pp. 29-30577 challenges from 79 competitions (2011-2025); CAISI conversion; category chart figure-only.
Software Development	extracted	pp. 31-34Table 4.8 all cells retained.
SWE-bench Verified	extracted	pp. 31-32489 problems from 12 repositories; excluded 11 original problems as Anthropic did; success requires all tests.
Breakpoint	extracted	pp. 33-34200-task subset; code-corruption framework; success requires all tests.
Science and Knowledge	extracted	pp. 34-39Table 4.13 all cells retained, source internal conflicts quarantined.
MMLU-Pro	extracted	p. 35Same 1,000 randomly sampled science-discipline questions for all models; not full 12,032-question benchmark.
MMMLU	extracted	p. 36Same 1,400 questions: 100 per 14 languages; all disciplines.
GPQA	extracted	p. 37GPQA-Diamond 198 questions; general knowledge score only; no biological task details.
HealthBench	extracted	p. 385,000 conversations; 48,562 rubric criteria; GPT-4.1 grader; rubric score cannot be treated as binary solves.
Humanity’s Last Exam	extracted	p. 392,158 text-only questions, not full 2,500 question/multimodal dataset.
Mathematical Reasoning	extracted	pp. 40-41Table 4.19 all cells retained; o4-mini equivalence grader.
Recent mathematics competitions	extracted	pp. 40-41SMT 58; OTIS-AIME 30; PUMaC 55 text-only tasks.
Cost Efficiency Analysis	extracted	pp. 42-44All Table 5.3 cells; curves not digitized; geometric expense ratios over overlapping achievable performance range.
Security Evaluations	qualitative	pp. 45-52Hijacking vs refusal/detail scoring separated; not complete security assessment.
Agent Hijacking	extracted	pp. 45-4830 benign tasks x five samples x two fixed injections, three malicious task classes; attempted actions conditional on injection exposure; default AgentDojo enhanced by CAISI.
Jailbreaking	qualitative	pp. 48-52Harmful biology/violence dataset: methodological metadata only; no dangerous capability comparisons or task content. Non-biological 30 hacking +50 scam queries; 17 public attacks, separate selection and evaluation sets; three graders. No operational prompts copied.
Censorship Evaluations	extracted	pp. 53-55190 free responses; narrative-flag rubric and LLM judge; all Table 7.5 cells. Tables 7.1-7.3 illustrative methodology only.
Model Adoption	extracted	pp. 56-60All exact printed prose statistics retained; charts not digitized.
Model Downloads	extracted	pp. 56-57Hugging Face/ModelScope publicly archived counts; downloads do not equal users.
Model Usage via API	extracted	pp. 57-59OpenRouter family/variant aggregation and limited platform coverage retained.
Derivative Model Uploads	extracted	pp. 59-60Exact prose relative bound retained; plot points figure-only; modification does not establish original model risk/performance.
Disclaimer	qualitative	p. 61Preliminary, partial, time/version-specific; benchmark contamination and generalization caveats; no certification or endorsement.
Agent Design	extracted	pp. 62-63CAISI ReAct; tool formats optimized model-by-model; same benchmark environment and tools across models.
Agent Resource Budget	extracted	pp. 63-64500,000 weighted tokens: output 100%, unique input 23%, total input 2%; not raw output token ceiling.
Uncertainty Estimation	extracted	pp. 64-65Mixed-effects logistic regression, model fixed effects/task random effects; delta-method SE. Not labelled IRT; no raw per-cell counts inferred.
Model Parameters	extracted	p. 65Model-specific agentic/nonagentic modes captured; DeepSeek temperature 0.6/top_p 0.95; Opus agentic temperature 0.7.
Model Token Prices	extracted	p. 66All six Table A5.1 values; historical authoring-time quote.
Cost Ratio Calculation	extracted	pp. 66-67Expense curves estimated from transcripts; truncated at alternate budget trigger; geometric mean over common achievable performance interval.
Selection of Benchmarks	qualitative	pp. 67-68Lower-contamination/real-world relevance rationale; multimodal inputs excluded; metadata only for CBRN.
Comparison to Self-Reported Benchmark Scores	reused	pp. 68-69All nonblank cells retained; Self columns explicitly reused developer results; no origin revalidation; blank cells not zero.
Table 3.1: Capability overview	extracted	p. 7All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.1: Cyber performance	extracted	p. 24All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.8: Software performance	extracted	p. 31All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.13: Science and knowledge performance	extracted	p. 34All printed nonblank numeric cells retained; duplicated results identified as such.
Table 4.19: Mathematics performance	extracted	p. 40All printed nonblank numeric cells retained; duplicated results identified as such.
Table 5.3: Performance and cost ratios	extracted	p. 44All printed nonblank numeric cells retained; duplicated results identified as such.
Table 7.5: Narrative alignment	extracted	p. 55All printed nonblank numeric cells retained; duplicated results identified as such.
Table A5.1: Token prices	extracted	p. 66All printed nonblank numeric cells retained; duplicated results identified as such.
Table A8.1: CAISI/self comparison	reused	p. 69All printed nonblank numeric cells retained; duplicated results identified as such.
Table 7.1: Example bilingual questions	qualitative	p. 53Illustrative method, not a measured model score; no synthetic example promoted to an observation.
Table 7.2: Example narrative flags	qualitative	p. 53Illustrative method, not a measured model score; no synthetic example promoted to an observation.
Table 7.3: Worked narrative alignment illustration	qualitative	p. 54Illustrative method, not a measured model score; no synthetic example promoted to an observation.
Figure 3.2	figure-only	p. 8No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.3	figure-only	p. 10No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.4	figure-only	p. 11No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.5	figure-only	p. 13No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.6	figure-only	p. 14No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.7	figure-only	p. 17No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.8	qualitative	p. 19CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 3.9	qualitative	p. 19CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 3.10	figure-only	p. 19No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.11	figure-only	p. 20No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.12	figure-only	p. 21No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 3.13	figure-only	p. 22No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.2	figure-only	p. 24No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.3	figure-only	p. 26No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.4	figure-only	p. 26No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.5	figure-only	p. 28No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.6	figure-only	p. 28No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.7	figure-only	p. 30No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.9	figure-only	p. 32No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.10	figure-only	p. 32No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.11	figure-only	p. 33No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.12	figure-only	p. 34No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.14	figure-only	p. 35No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.15	figure-only	p. 36No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.16	figure-only	p. 37No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.17	figure-only	p. 38No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.18	figure-only	p. 39No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.20	figure-only	p. 40No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.21	figure-only	p. 41No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 4.22	figure-only	p. 41No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 5.1	figure-only	p. 42No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 5.2	figure-only	p. 43No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.1	figure-only	p. 45No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.2	figure-only	p. 47No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.3	figure-only	p. 48No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.4	figure-only	p. 49No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.5	figure-only	p. 49No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.6	qualitative	p. 51CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 6.7	qualitative	p. 51CBRN: method metadata and high-level limitations only; numerical dangerous-capability comparisons omitted.
Figure 6.8	figure-only	p. 52No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 6.9	figure-only	p. 52No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 7.4	figure-only	p. 55No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.1	figure-only	p. 56No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.2	figure-only	p. 57No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.3	figure-only	p. 58No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.4	figure-only	p. 59No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.5	figure-only	p. 59No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure 8.6	figure-only	p. 60No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.
Figure A1.1	figure-only	p. 62No bar or curve heights estimated; exact values extracted only where printed in accompanying tables or prose.

来源待核实事项

本报告使用 CVE-Bench、Cybench 和 CTF-Archive，未评测 CyberGym。Cybench 的实际方法位于 §4.1.2。

报告同时描述经验平均与混合效应估计；任务总数不能直接用来反推表中分数的成功次数。

版本指纹

document