{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"strongreject","title":"A StrongREJECT for Empty Jailbreaks","aliases":["StrongREJECT"],"authors":["Alexandra Souly","Qingyuan Lu","Dillon Bowen","Tu Trinh","Elvis Hsieh","Sana Pandey","Pieter Abbeel","Justin Svegliato","Scott Emmons","Olivia Watkins","Sam Toyer"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"strongreject","attributionSourceId":"strongreject-author-source","originalUrl":"https://arxiv.org/abs/2402.10260v2","publicationDate":"2024-02-15","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"General safety evidence retained for context; no defensible direct link to the six NUWA risk entries was found.","zh":"保留为一般安全参考；当前无法据来源建立与六个 NUWA 风险入口的直接关联。"},"protocolIds":["strongreject"],"contributions":[{"organizationId":"berkeley","role":"research","sourceId":"strongreject-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"The informativeness of responses to prohibited requests.","zh":"对禁止请求所作回答的信息效用。"},"note":{"question":{"en":"A graded response evaluator for jailbreak effectiveness.","zh":"用于越狱有效性分析的回答评分方法。"},"method":{"en":"Model responses graded for jailbreak effectiveness","zh":"按越狱效果评分的模型响应"},"finding":{"en":"The informativeness of responses to prohibited requests.","zh":"对禁止请求所作回答的信息效用。"},"supports":null,"limits":{"en":"A response score is not an end-to-end measure of harmful action.","zh":"回答得分不是端到端有害行动测量。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"strongreject","title":"A StrongREJECT for Empty Jailbreaks","en":{"question":"Do jailbreak evaluators measure useful harmful content or merely willingness to answer?","design":"Curated forbidden questions and rubric/fine-tuned graders are compared against human judgments, emphasizing specificity and usefulness of responses.","findings":["Existing evaluators frequently overstated jailbreak effectiveness.","Correlation with human jailbreak rankings remained above 0.84 across the reported robustness permutations (Appendix D)."],"interpretation":"NUWA interpretation: apparent attack gains may reflect permissive scoring rather than preserved harmful-task capability.","limits":"Small numbers of labeled jailbreaks limit some per-model correlations; the work does not validate multimodal grading.","decisionRelevance":"Model recommendation: validate the evaluator on human-rated outputs before adopting reported attack-success claims.","takeaway":"NUWA interpretation: apparent attack gains may reflect permissive scoring rather than preserved harmful-task capability.","connections":[],"scoring":"Evaluator measures response specificity/usefulness rather than nonrefusal alone; checked against human judgments.","conditions":null,"statistics":"Correlation with human jailbreak rankings remained above 0.84 across the reported robustness permutations (Appendix D).","usage":"Model recommendation: validate the evaluator on human-rated outputs before adopting reported attack-success claims."},"zh":{"question":"越狱评估器测量的是有用有害内容，还是仅回答意愿？","design":"精选禁答问题与量表、微调判分器对照人工判断，强调回答的具体性和有用性。","findings":["既有评估器经常高估越狱效果。","所报稳健性组合中，与人类攻击排序的相关均超过0.84（附录D）。"],"interpretation":"NUWA 解读：表面攻击提升可能来自宽松判分，而非保留有害任务能力。","limits":"部分模型人工标注攻击较少，限制相关估计；尚未验证多模态判分。","decisionRelevance":"模型建议：采用攻击成功结论前，先以人工评分输出验证评估器。","takeaway":"NUWA 解读：表面攻击提升可能来自宽松判分，而非保留有害任务能力。","connections":[],"scoring":"评价具体性与有用性，而非仅不拒绝，并与人工判断核对。","conditions":null,"statistics":"所报稳健性组合中，与人类攻击排序的相关均超过0.84（附录D）。","usage":"模型建议：采用攻击成功结论前，先以人工评分输出验证评估器。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"strongreject-author-source","url":"https://arxiv.org/html/2402.10260v2","locator":"§2 The StrongREJECT benchmark","version":"Selected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established","artifactSha256":"ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71"},{"sourceId":"strongreject-author-source","url":"https://arxiv.org/html/2402.10260v2","locator":"§3 Human evaluation","version":"Selected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established","artifactSha256":"ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71"},{"sourceId":"strongreject-author-source","url":"https://arxiv.org/html/2402.10260v2","locator":"Appendix D Robustness of Spearman correlation results","version":"Selected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established","artifactSha256":"ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71"},{"sourceId":"strongreject-author-source","url":"https://arxiv.org/html/2402.10260v2","locator":"§5 Discussion","version":"Selected original resource https://arxiv.org/html/2402.10260v2; sha256:ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71; archived source capture; exact capture time not independently established","artifactSha256":"ffbcbe49cc39b51c0cd776d0cab7d8d30579afd0e7db208e047c5a8ad1dd9b71"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"898b32b1081a806794fea4caf6b48f34cb852bdafa6feee1f7bc909b14b2656f"}},"sources":[{"id":"strongreject","title":"A StrongREJECT for Empty Jailbreaks","url":"https://arxiv.org/abs/2402.10260v2","version":"arXiv:2402.10260v2","publicationDate":"2024-08-27","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"arxiv:2402.10260","authors":["Alexandra Souly ∗","Qingyuan Lu ∗","Dillon Bowen ∗","Tu Trinh †","Elvis Hsieh †","Sana Pandey","Pieter Abbeel","Justin Svegliato","Scott Emmons ‡","Olivia Watkins ‡","Sam Toyer ‡","Center for Human-Compatible AI","UC Berkeley"],"aliases":["https://doi.org/10.48550/arXiv.2402.10260","https://arxiv.org/abs/2402.10260","https://arxiv.org/pdf/2402.10260"]}]}