Risk Evaluation · Benchmark

A StrongREJECT for Empty Jailbreaks

NUWA research note · independent review pending

First published: 2024-02-15Source checked: 2026-10-25

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: apparent attack gains may reflect permissive scoring rather than preserved harmful-task capability.

Research question

Do jailbreak evaluators measure useful harmful content or merely willingness to answer?

Design and assumptions

Curated forbidden questions and rubric/fine-tuned graders are compared against human judgments, emphasizing specificity and usefulness of responses.

StrongREJECT — Figure 2 — Figure 2 · Original result figure. Interpret within the protocol scope and limitations below. Original figure and source ↗

Source findings

Existing evaluators frequently overstated jailbreak effectiveness.

Correlation with human jailbreak rankings remained above 0.84 across the reported robustness permutations (Appendix D).

NUWA interpretation

NUWA interpretation: apparent attack gains may reflect permissive scoring rather than preserved harmful-task capability.

Limits and contrary evidence

Small numbers of labeled jailbreaks limit some per-model correlations; the work does not validate multimodal grading.

Research and decision relevance

Model recommendation: validate the evaluator on human-rated outputs before adopting reported attack-success claims.

Precise source locations

A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · §2 The StrongREJECT benchmarkSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established
A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · §3 Human evaluationSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established
A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · Appendix D Robustness of Spearman correlation resultsSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established
A StrongREJECT ¯ \overline{\hbox{{StrongREJECT}}} for Empty Jailbreaks · §5 DiscussionSelected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by University of California, Berkeley

Authors (11)

Alexandra Souly · Qingyuan Lu · Dillon Bowen · Tu Trinh · Elvis Hsieh · Sana Pandey · Pieter Abbeel · Justin Svegliato · Scott Emmons · Olivia Watkins · Sam Toyer

Evaluation protocols

StrongREJECT →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{strongreject,
  title = {A StrongREJECT for Empty Jailbreaks},
  url = {https://arxiv.org/abs/2402.10260v2},
  author = {Alexandra Souly and Qingyuan Lu and Dillon Bowen and Tu Trinh and Elvis Hsieh and Sana Pandey and Pieter Abbeel and Justin Svegliato and Scott Emmons and Olivia Watkins and Sam Toyer},
  year = {2024},
  date = {2024-02-15},
  version = {arXiv:2402.10260v2},
  note = {arXiv:2402.10260v2}
}

NUWA record · BibTeX

@misc{strongreject-nuwa,
  title = {A StrongREJECT for Empty Jailbreaks — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/strongreject/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--strongreject.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--strongreject.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Alexandra Souly ∗, Qingyuan Lu ∗, et al.. A StrongREJECT for Empty Jailbreaks. 2024-08-27. v2 [arXiv]; Author affiliations on the original paper title page; §2 The StrongREJECT benchmark; §3 Human evaluation