{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"caisi-deepseek-2025-report","title":"Evaluation of DeepSeek AI Models","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"technical-report","sourceId":"caisi-deepseek-2025-source","attributionSourceId":"caisi-deepseek-2025-source","originalUrl":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf","publicationDate":"2025-09-30","updatedOn":"2026-10-25","areas":["autonomy","deception","cybersecurity","cbrn","disinformation"],"scopeNote":{"en":"Reported results retain their original evaluator, configuration and measurement scope. NUWA organizes and interprets the evidence; it has not independently reproduced these experiments.","zh":"结果保留原始执行方、配置与测量范围。NUWA 整理和解读证据，未独立复现这些实验。"},"protocolIds":["cybench","cybergym","agentdojo"],"contributions":[{"organizationId":"nist","role":"research","sourceId":"caisi-deepseek-2025-source","locator":"Report masthead and stated evaluation methodology"}],"summary":{"en":"Reported differences depend on task, scaffold and scoring; institutional judgments are retained as source judgments rather than NUWA findings.","zh":"所报告差异依赖任务、脚手架与评分；机构判断保留为来源观点，不改写为 NUWA 实验发现。"},"note":{"question":{"en":"How do selected DeepSeek releases perform under CAISI’s capability and security protocols?","zh":"选定 DeepSeek 版本在 CAISI 能力与安全协议下表现如何？"},"method":{"en":"The September 2025 report covers public and held-out tasks, inference configuration, costs, hijacking and jailbreak evaluations.","zh":"2025 年 9 月报告覆盖公开与保留任务、推理配置、费用、劫持与越狱评测。"},"finding":{"en":"Reported differences depend on task, scaffold and scoring; institutional judgments are retained as source judgments rather than NUWA findings.","zh":"所报告差异依赖任务、脚手架与评分；机构判断保留为来源观点，不改写为 NUWA 实验发现。"},"supports":null,"limits":{"en":"Controlled tasks, elicitation, benchmark familiarity and configuration differences constrain generalization to deployment.","zh":"受控任务、诱发方式、基准熟悉程度和配置差异限制对部署的外推。"},"connections":null},"resources":[],"tags":[]},"reading":{"id":"caisi-deepseek-2025-report","title":"Evaluation of DeepSeek AI Models","en":{"question":"How do selected DeepSeek releases compare on controlled capability and security evaluations?","design":"The September 2025 report evaluates R1, R1-0528 and V3.1 with US reference models across nineteen public/private benchmarks, with agent interfaces adapted per model.","findings":["The report finds the best US reference ahead of the best DeepSeek release on nearly every tested benchmark, with larger cyber/software gaps.","It separately studies capability, cost efficiency, security, censorship and adoption rather than treating them as one metric."],"interpretation":"NUWA interpretation: independent benchmark selection and explicit budget conventions are essential to cross-provider interpretation.","limits":"These are dated releases and selected tasks; cost efficiency uses an expense–performance method, not token prices alone, and uncertainty labels include standard errors.","decisionRelevance":"Model recommendation: cite benchmark-specific budgets, model releases and uncertainty units; do not convert national comparisons into an undated global ranking.","takeaway":"NUWA interpretation: independent benchmark selection and explicit budget conventions are essential to cross-provider interpretation.","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"Model recommendation: cite benchmark-specific budgets, model releases and uncertainty units; do not convert national comparisons into an undated global ranking."},"zh":{"question":"受控能力及安全评测中，所选DeepSeek版本表现如何？","design":"2025年9月报告比较R1、R1-0528、V3.1及美国参考模型，覆盖19项公开私有基准，并逐模型适配接口。","findings":["所测基准中最佳美国参考几乎均领先最佳DeepSeek，网络与软件差距较大。","报告分开能力、成本、安全、审查及采用，而非单一指标。"],"interpretation":"NUWA 解读：独立基准选择及预算约定对跨提供方解释至关重要。","limits":"版本和任务均有时点性，成本以支出表现曲线而非仅令牌价格，不确定性含标准误。","decisionRelevance":"模型建议：逐基准引用预算、版本及误差单位，不变为无日期全球排名。","takeaway":"NUWA 解读：独立基准选择及预算约定对跨提供方解释至关重要。","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"模型建议：逐基准引用预算、版本及误差单位，不变为无日期全球排名。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"caisi-deepseek-2025-source","url":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf","locator":"PDF p.2 §1 Executive Summary","version":"Selected original resource https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf; sha256:4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af; archived source capture; exact capture time not independently established","artifactSha256":"4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af"},{"sourceId":"caisi-deepseek-2025-source","url":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf","locator":"PDF pp.4–5 §2 Methodology","version":"Selected original resource https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf; sha256:4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af; archived source capture; exact capture time not independently established","artifactSha256":"4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af"},{"sourceId":"caisi-deepseek-2025-source","url":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf","locator":"PDF p.63 Appendix: agent configuration","version":"Selected original resource https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf; sha256:4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af; archived source capture; exact capture time not independently established","artifactSha256":"4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af"},{"sourceId":"caisi-deepseek-2025-source","url":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf","locator":"PDF p.66 A5 Model Token Prices and A6 Cost Ratio Calculation","version":"Selected original resource https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf; sha256:4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af; archived source capture; exact capture time not independently established","artifactSha256":"4b267893d3c4b6b48d36deb99efec591c414d2e2608be0625d90b477298c82af"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"4d6e76f5a1a7a448c5aaa98585b5e5048e7b326d86419b7f48e717d047d14a6b"}},"sources":[{"id":"caisi-deepseek-2025-source","title":"Evaluation of DeepSeek AI Models","url":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf","version":"30 September 2025","publicationDate":"2025-09-30","datePrecision":"day","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"https://www.nist.gov/system/files/documents/2025/09/30/CAISI_Evaluation_of_DeepSeek_AI_Models.pdf"}]}