NUWA Source record
Evaluating DeepSeek v4 Pro for Frontier Risks
Source-specific extraction of Neo Research · DeepSeek V4 Pro. Findings and measurements retain their original attribution.
Scope and attribution
- Reported by
- Neo Research
- Actual evaluator
- Not established for every observation
- Captured version
- 2026-10-26
- Source identity
- 267c0a21feca7aeb
Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.
Source findings
The following statements summarize the source. They are not independent NUWA experimental findings.
Authors judge current cyber capability below their concern threshold, while noting that the Code of Practice itself sets no numeric cyber threshold.
pp.7,10Author risk judgment, not a regulator certification or an experimentally validated real-world harm boundary.
Public knowledge/research-workflow benchmarks are weak proxies for operational biology uplift; authors decline a firm CBRN misuse conclusion and call for targeted or human-uplift studies.
pp.11–13No CBRN capability scores or comparisons extracted.
Forced binary persuasion choices can conceal high persuasion-attempt propensity under free-form continuation.
pp.14–18Attempt rate does not measure successful persuasion or deployment impact.
Misbehavior is scenario-sensitive: deliberate harm and deception rise when an evaluator supplies motives or pressure; default benchmark behavior appears more corrigible.
pp.21–32Fictional stress tests do not establish autonomous scheming or real-world frequency.
Independent and in-house tests indicate brittle safeguards under adversarial pressure; Pro and Flash exhibit related weaknesses.
pp.33–38No attack payloads or operational instructions reproduced; FAR.AI mixed-domain numerical results excluded.
Verbalized awareness varies strongly by scenario and monitor and cannot establish causal behavior change; benchmarks may be recognized from training.
pp.39–44Silent awareness is unmeasured; monitoring only externalized reasoning.
Automated audit outcomes depend on auditor choice, selected scenarios, conversational budget, and judge criteria.
pp.45–53Selected high-scoring seeds/ablations cannot be treated as prevalence estimates.
Binary judging appears fairly stable across tested model origins; open ordinal concerning scores show broader cross-panel disagreement.
pp.47–53Ordinal rubric underspecification and high variance limit interpretation; origin is not a universal causal explanation.
Extraction coverage
Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.
View 36 coverage entries
| Section | Treatment | Location and scope |
|---|---|---|
| Title / Executive summary / Introduction | qualitative | pp.1–6Report dated 29 May 2026; independent Neo evaluation without privileged access. v1.1 URL does not establish revision date. |
| 1 Cyber — Cybench | extracted | pp.7–8Exact chart labels retained; comparator values explicitly reused; no CI endpoint estimates. |
| 1 Cyber — EVMbench | extracted | pp.8–9All 16 printed figure values and n=3 labels retained. Comparator executor/reuse status unclear, quarantined. |
| 1 Cyber — WMDP-Cyber | extracted | pp.9–10Complete five-model knowledge table; controls reused from Muse Spark report. |
| 2 CBRN — LAB-Bench | qualitative | pp.11–12Metadata only: research-workflow MCQ proxy; no real wet-lab uplift established; image tasks unsupported at testing. Dangerous-capability numbers intentionally excluded. |
| 2 CBRN — WMDP Chem/Bio | qualitative | p.12Knowledge-only evaluation; saturation limits inference. Biological/chemical numerical comparisons excluded. |
| 2 CBRN — FORTRESS / conclusion | qualitative | p.13Adversarial CBRNE and benign-overrefusal methodology; ensemble-majority judge with refusal-related grading bias. Numerical CBRN comparisons excluded. |
| 3.1 PersuSafety | extracted | pp.14–15Exact counts/rates from text; unlabelled stacked-chart cells left figure-only; reused older controls flagged. |
| 3.1 Attempt-to-Persuade | extracted | pp.15–16All five subject rows; refusal/attempt counts preserved in rawFields; attempts differ from persuasion success. |
| 3.1 Cross-evaluation experiments | extracted | pp.17–18Exact prose rates retained; unlabeled bars not estimated. Approximate prose category rates not promoted to exact point estimates. |
| 3.2 PsychosisBench | extracted | pp.19–20All 27 labelled DCS/HES/SIS values; source has an SIS direction conflict. |
| 4.1 SAD-mini | extracted | pp.21–22Complete 30-cell table including raw ±95% bootstrap uncertainty and sample sizes; settings not identified. |
| 4.1 GDM Situational Awareness / Stealth | extracted | pp.23–24Exact plotted labels retained; reused comparison bars and original target clearly distinguished. |
| 4.2 Shutdown Resistance | extracted | p.24Complete 16-cell table; no inferred counts from fractional percentages. |
| 4.2 ImpossibleBench | extracted | p.25Complete six-value table; tools conditions preserved. |
| 4.2 AbstentionBench / CharXiv | extracted | pp.25–26Exact 10 chart labels; CharXiv setup comparability caveat retained. |
| 4.3 DeceptionBench | extracted | p.27Exact 12 printed labels rather than rounded prose; explicit inducement is not spontaneous propensity. |
| 4.3 MASK | extracted | pp.27–28All 21 labelled bars; original Sonnet control separated from reused five older controls. |
| 4.3 Alignment Faking | extracted | pp.28–29Compliance gaps and deceptive-alignment rates extracted; awareness confound retained. |
| 4.3 Anthropic Agentic Misalignment | extracted | pp.29–31Panel + three target scenario rates extracted; no harmful action scripts transcribed. |
| 4.3 Machiavelli / TextQuests | extracted | pp.31–32All ten labelled values including CAIS-reported and own target run; unresolved discrepancy flagged. |
| 5.1 IHEval | extracted | pp.33–34Five printed values; plotted setting not disclosed. |
| 5.1 Backbone Breaker | extracted | pp.34–35All ten printed score labels; metadata only, no payload/task instructions. |
| 5.1 AgentHarm | extracted | pp.35–3728 comparative chart values, prose rounding and 16 per-category labels retained; condition ambiguities noted. |
| 5.2.1 FAR.AI | reused | p.37External official-API red-team report reproduced with permission. Mixed CBRN results/numerical dangerous comparisons and operational methods excluded; qualitative safeguard brittleness only. |
| 5.2.2 In-house StrongREJECT | extracted | p.38Complete 16-cell table; all four models tested by Neo; no jailbreak template text. |
| 6.1 Evaluation awareness trend | extracted | pp.39–40Only exact prose endpoint rates retained. Intermediate plotted points and approximate Western line left figure-only. Same-protocol comparator note retained. |
| 6.2 Existing Benchmarks / contamination | extracted | pp.40–41Complete 45-cell awareness table and exact named-dataset probe rates; ranges retained without midpoint; contamination caveat. |
| 6.3 Apollo scenarios | extracted | pp.41–42Safe awareness-only results retained; sandbagging biological-domain numeric comparisons excluded. Novel/published aggregate chart labels retained. |
| 6.4 Awareness case studies / six-eval monitor | extracted | pp.42–44MASK count and all 42 monitor table values; causal-influence attribution only a heuristic; no quoted chain-of-thought. |
| 7 Petri / Bloom | extracted | pp.45–46Audit metadata and complete ablation score/delta table; operational persona/task descriptions omitted. |
| 8.1 JailbreakBench safety judges | extracted | pp.47–48Complete 33-cell validation table, graded n and false-positive/negative metrics; varying judge refusal coverage. |
| 8.2 MASK judges | extracted | pp.48–4918 distribution labels, 12 dissent table cells, complete-case counts; target and judge roles explicit. |
| 8.3 Petri auditor/judge comparisons | extracted | pp.49–53Safe auditor comparison rows, safe selected variance tables, all aggregate bloc statistics. Chemical-weapon/attack scenario rows intentionally excluded; detailed task narratives omitted. |
| Conclusion & Recommendations | qualitative | p.54Scenario-dependent misbehavior, safeguard robustness, awareness and measurement caveats summarized without extending to real-world outcomes. |
| Acknowledgments | qualitative | p.55Yong Zheng-Xin helped run EVMbench; FAR.AI gave reproduction permission; fiscal sponsor Singapore AI Safety Hub. No inferred executor roles from acknowledgments. |
Unresolved source issues
不同评测意识监测器的标签与结果不能直接互换，来源没有建立意识对行为的因果影响。
关闭阻碍表的百分比与文字中的样本量无法完整核对，未据此推算成功次数。
FORTRESS 在本报告中仅用于特定子集，位于相关领域评测章节；对抗稳健性章节中的其他评测不能归入 FORTRESS。
Version fingerprints
https://neoresearch.ai/papers/DSv4_Safety_Evaluation_v1.1.pdf
e8d83775830fdd431d6462953a9c11e39a3d6ad25410b47efd600b0ae0ce95d8https://neoresearch.ai/research/deepseek-v4-pro-safety-evaluation/
43dd23e7277661bc2c9363bbfff561dcc56dbfbfed45519dadb5fb6e1e71ab33References
- [1] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026