Evaluation dossier

StrongREJECT

The informativeness of responses to prohibited requests.

University of California, Berkeley / arXiv:2402.10260v2 / Behavior

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

StrongREJECT · Baseline StrongREJECT

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

4 observations · 0 dated

● Matched◇ Configuration differs□ Some conditions undisclosed

4 observations lack a verified model release date. They remain in Data and Systems.

Select a point or table row to inspect its conditions and provenance.

4 observations

Measurements and source conditions
System / model	Result	Condition status	Task / conditions	Evaluator / report	Dates
DeepSeek v4 Flash	0.04 scoreBaseline StrongREJECT	Incomplete	StrongREJECTBaseline StrongREJECT—	Not disclosedEvaluating DeepSeek v4 Pro for Frontier Risks	Release —Report 2026-05-29
DeepSeek v4 Pro	0.03 scoreBaseline StrongREJECT	Incomplete	StrongREJECTBaseline StrongREJECT—	Not disclosedEvaluating DeepSeek v4 Pro for Frontier Risks	Release —Report 2026-05-29
Kimi K2.6	0.02 scoreBaseline StrongREJECT	Incomplete	StrongREJECTBaseline StrongREJECT—	Not disclosedEvaluating DeepSeek v4 Pro for Frontier Risks	Release —Report 2026-05-29
Qwen3.6-Max (preview)	0.02 scoreBaseline StrongREJECT	Incomplete	StrongREJECTBaseline StrongREJECT—	Not disclosedEvaluating DeepSeek v4 Pro for Frontier Risks	Release —Report 2026-05-29

Task and measurement

Benchmark protocol profile

Task / subject: Model responses graded for jailbreak effectiveness
Measurement target: A graded response evaluator for jailbreak effectiveness.
Scoring / outcome: Baseline jailbreak rate · Source-reported measurement under the recorded configuration.
Interpretation boundary: A response score is not an end-to-end measure of harmful action.

A reported evaluation

The source evaluates DeepSeek v4 Pro · StrongREJECT.

Reported measurement: Baseline jailbreak rate · 0.6%.

Source-reported measurement under the recorded configuration. Source denominator: 313 (unit not disclosed)

Baseline jailbreak rate: 0.6 percent. Conditions are retained from the source.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: Do jailbreak evaluators measure useful harmful content or merely willingness to answer?
Task population: Curated forbidden questions and rubric/fine-tuned graders are compared against human judgments, emphasizing specificity and usefulness of responses.
Scoring and failure modes: Evaluator measures response specificity/usefulness rather than nonrefusal alone; checked against human judgments.
System conditions: Not established by the retained material
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: Correlation with human jailbreak rankings remained above 0.84 across the reported robustness permutations (Appendix D).
Validity and interpretation: Small numbers of labeled jailbreaks limit some per-model correlations; the work does not validate multimodal grading.
Appropriate research use: Model recommendation: validate the evaluator on human-rated outputs before adopting reported attack-success claims.

A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · §2 The StrongREJECT benchmarkSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established
A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · §3 Human evaluationSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established
A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · Appendix D Robustness of Spearman correlation resultsSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established
A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · §5 DiscussionSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

Neo Research

Evaluating DeepSeek v4 Pro for Frontier Risks →

Actual evaluators: Not disclosed

16 result records · 4 reported system names

Protocols and comparison decisions (4)

StrongREJECT: Attack jailbreak rate · Neo Researchsource-report · Scope reviewed

StrongREJECT: Attack StrongREJECT · Neo Researchsource-report · Scope reviewed

StrongREJECT: Baseline jailbreak rate · Neo Researchsource-report · Scope reviewed

StrongREJECT: Baseline StrongREJECT · Neo Researchsource-report · Scope reviewed

Original source charts and materials

StrongREJECT — Figure 2 — **Figure 2** · Original result figure. Interpret within the protocol scope and limitations below. View original image ↗

References

[1] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 38; StrongREJECT; Baseline jailbreak rate; PDF p. 38; StrongREJECT; Attack jailbreak rate; PDF p. 38; StrongREJECT; Baseline StrongREJECT
[2] Alexandra Souly ∗, Qingyuan Lu ∗, et al.. A StrongREJECT for Empty Jailbreaks. 2024-08-27. v2 [arXiv]; Author affiliations on the original paper title page; Figure 2

Search NUWA Index