Evaluation dossier

WMDP

Hazardous knowledge proxies and unlearning evaluation.

Center for AI Safety · University of California, Berkeley · Massachusetts Institute of Technology · SecureBio · Scale AI · New York University · IIIT Hyderabad · Stanford University · Harvard University · University of Southern California · University of Illinois Urbana-Champaign · Lapis Labs · University of California, Los Angeles · California Institute of Technology · Sybil · Northeastern University · Carnegie Mellon University · RTX BBN Technologies · University of Newcastle · Keio University · Ariel University · Arizona State University · xAI / arXiv:2403.03218v7 / Capability

Sources and provenance ↓ Read the research narrative →

Task and measurement

Benchmark protocol profile

Task / subject: Model answering hazardous knowledge questions
Measurement target: Multiple-choice knowledge measurements.
Scoring / outcome: WMDP Bio accuracy · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability.
Interpretation boundary: Knowledge is a proxy; scores do not establish practical harmful-task ability.

A reported evaluation

The source evaluates zephyr-7b · Table 1 · WMDP.

Reported measurement: WMDP Bio accuracy · 63.7%.

Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. See the source protocol; no independent trial count inferred.

Source-reported measurements for zephyr-7b in Table 1; values are transcribed, not independently reproduced.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: Can a public proxy benchmark track hazardous knowledge while supporting mitigation research?
Task population: WMDP contains 3,668 expert-written multiple-choice items across three risk domains, filtered to exclude sensitive and export-controlled information.
Scoring and failure modes: Multiple-choice hazardous-knowledge proxy accuracy plus separate general-capability retention.
System conditions: 3,668 filtered expert questions; no operational task execution is measured.
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: Reported mitigation gains also affect some adjacent benign subjects, revealing a precision trade-off (§7).
Validity and interpretation: A low multiple-choice score does not by itself establish durable removal of knowledge or reduced real-world misuse.
Appropriate research use: Model recommendation: evaluate safeguard durability and benign utility separately; retain only nonoperational benchmark statistics in this index.

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning · AbstractSelected original resource https://arxiv.org/html/2403.03218v7; sha256:472585ab0b6df3fe4f74bee849a6a68087e4641617a5c8914c0b057a27ca3d78; archived source capture; exact capture time not independently established
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning · §3 The WMDP BenchmarkSelected original resource https://arxiv.org/html/2403.03218v7; sha256:472585ab0b6df3fe4f74bee849a6a68087e4641617a5c8914c0b057a27ca3d78; archived source capture; exact capture time not independently established
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning · §5 Experimental ResultsSelected original resource https://arxiv.org/html/2403.03218v7; sha256:472585ab0b6df3fe4f74bee849a6a68087e4641617a5c8914c0b057a27ca3d78; archived source capture; exact capture time not independently established
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning · §7 ConclusionSelected original resource https://arxiv.org/html/2403.03218v7; sha256:472585ab0b6df3fe4f74bee849a6a68087e4641617a5c8914c0b057a27ca3d78; archived source capture; exact capture time not independently established

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

Actual evaluators: Not disclosed

9 result records · 9 reported system names

Protocols and comparison decisions (0)

Neo Research

Evaluating DeepSeek v4 Pro for Frontier Risks →

Actual evaluators: Neo Research · Meta

5 result records · 5 reported system names

Protocols and comparison decisions (0)

Original source charts and materials

Reported results

Source order · reported configurations

WMDP Bio accuracy

0–100%

zephyr-7b

zephyr-7b · 63.7% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

63.7%

zephyr-7b · LLMU

zephyr-7b · LLMU · 59.5% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

59.5%

zephyr-7b · SCRUB

zephyr-7b · SCRUB · 43.8% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

43.8%

zephyr-7b · SSD

zephyr-7b · SSD · 50.2% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

50.2%

zephyr-7b · RMU

zephyr-7b · RMU · 31.2% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

31.2%

Yi-34b

Yi-34b · 75.3% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

75.3%

Yi-34b · RMU

Yi-34b · RMU · 30.7% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

30.7%

Mixtral-8x7B

Mixtral-8x7B · 74.8% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

74.8%

Mixtral-8x7B · RMU

Mixtral-8x7B · RMU · 34.0% · Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. · See the source protocol; no independent trial count inferred.

34.0%

Showing 9 of 9 systems

Values describe the measured outcome under the source protocol, not a general safety score.

Table 1: WMDP biology and cyber knowledge accuracy for base models and unlearning variants; 25% is random choice. Knowledge tests do not establish end-to-end harmful capability. Evaluation date not specified; publication / record dates are shown below.

Evaluation conditions and system configurations

zephyr-7b: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

zephyr-7b · LLMU: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

zephyr-7b · SCRUB: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

zephyr-7b · SSD: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

zephyr-7b · RMU: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Yi-34b: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Yi-34b · RMU: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Mixtral-8x7B: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Mixtral-8x7B · RMU: See the source protocol; no independent trial count inferred. · Table 1 · WMDP; modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Reported results · source order
System / configuration	WMDP Bio accuracy	WMDP Cyber accuracy	Publication / verification
zephyr-7bHarness not specified	63.7%	44.0%	Source checked2026-10-25
zephyr-7b · LLMUHarness not specified	59.5%	39.5%	Source checked2026-10-25
zephyr-7b · SCRUBHarness not specified	43.8%	39.3%	Source checked2026-10-25
zephyr-7b · SSDHarness not specified	50.2%	35.0%	Source checked2026-10-25
zephyr-7b · RMUHarness not specified	31.2%	28.2%	Source checked2026-10-25
Yi-34bHarness not specified	75.3%	49.7%	Source checked2026-10-25
Yi-34b · RMUHarness not specified	30.7%	29.0%	Source checked2026-10-25
Mixtral-8x7BHarness not specified	74.8%	52.0%	Source checked2026-10-25
Mixtral-8x7B · RMUHarness not specified	34.0%	30.8%	Source checked2026-10-25

Assessment provenance

Evaluating DeepSeek v4 Pro for Frontier Risks → · External report

References

[1] Nathaniel Li ∗, Alexander Pan ∗, et al.. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. 2024-05-15. v7 [arXiv]; Author affiliations on the original paper title page; Table 1
[2] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 9; WMDP-Cyber; accuracy

Search NUWA Index