{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"harmbench","title":"HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal","aliases":["HarmBench"],"authors":["Mantas Mazeika","Long Phan","Xuwang Yin","Andy Zou","Zifan Wang","Norman Mu","Elham Sakhaee","Nathaniel Li","Steven Basart","Bo Li","David Forsyth","Dan Hendrycks"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"harmbench","attributionSourceId":"harmbench-author-source","originalUrl":"https://arxiv.org/abs/2402.04249v2","publicationDate":"2024-02-06","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"General safety evidence retained for context; no defensible direct link to the six NUWA risk entries was found.","zh":"保留为一般安全参考；当前无法据来源建立与六个 NUWA 风险入口的直接关联。"},"protocolIds":["harmbench"],"contributions":[{"organizationId":"uiuc","role":"research","sourceId":"harmbench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"cais","role":"research","sourceId":"harmbench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"cmu","role":"research","sourceId":"harmbench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"berkeley","role":"research","sourceId":"harmbench-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"microsoft","role":"research","sourceId":"harmbench-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Robust refusal under automated red teaming.","zh":"自动化红队测试下的拒答稳健性。"},"note":{"question":{"en":"Attack-dependent generation of harmful responses.","zh":"依赖具体攻击方法的有害回答生成情况。"},"method":{"en":"Model responses under adversarial prompting","zh":"对抗提示下的模型响应"},"finding":{"en":"Robust refusal under automated red teaming.","zh":"自动化红队测试下的拒答稳健性。"},"supports":null,"limits":{"en":"Evaluator and attack coverage affect scores; refusal alone does not establish safety.","zh":"评判器与攻击覆盖影响分数；拒答本身不能证明安全。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"harmbench","title":"HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal","en":{"question":"Can automated red-team methods and defenses be compared with consistent behavior-level scoring?","design":"The framework standardizes harmful behaviors, generation and classification; its reported study compares eighteen attacks and 33 target models/defenses.","findings":["Contextual behaviors had higher attack success than some other functional categories.","Copyright scoring uses a stricter content-matching criterion than the noncopyright classifiers (§6.1)."],"interpretation":"NUWA interpretation: attack-success rates depend on both behavior mix and the success classifier.","limits":"Functional categories use different decision criteria; pooled success cannot be read as an equivalent real-world-harm scale.","decisionRelevance":"Model recommendation: retain the classifier and category breakdown when comparing attacks or safety interventions.","takeaway":"NUWA interpretation: attack-success rates depend on both behavior mix and the success classifier.","connections":[],"scoring":"Classifier-determined attack-success rates; functional categories use different criteria.","conditions":"The reported study compares eighteen attack methods and 33 targets/defenses.","statistics":"Copyright scoring uses a stricter content-matching criterion than the noncopyright classifiers (§6.1).","usage":"Model recommendation: retain the classifier and category breakdown when comparing attacks or safety interventions."},"zh":{"question":"自动红队方法与防护能否采用统一的行为级判分比较？","design":"框架标准化有害行为、生成及分类；所报研究比较18种攻击与33个目标模型或防护。","findings":["上下文类行为的攻击成功率较高。","版权类采用比非版权分类器更严格的内容匹配判据（§6.1）。"],"interpretation":"NUWA 解读：攻击成功率同时取决于行为组成和成功分类器。","limits":"功能类别判据不同，汇总成功率不能视为等价的真实伤害尺度。","decisionRelevance":"模型建议：比较攻击或防护时保留分类器及类别明细。","takeaway":"NUWA 解读：攻击成功率同时取决于行为组成和成功分类器。","connections":[],"scoring":"分类器判攻击成功，各功能类别判据不同。","conditions":"所报研究比较18攻击与33目标或防护。","statistics":"版权类采用比非版权分类器更严格的内容匹配判据（§6.1）。","usage":"模型建议：比较攻击或防护时保留分类器及类别明细。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"harmbench-author-source","url":"https://arxiv.org/html/2402.04249v2","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2402.04249v2; sha256:c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d; archived source capture; exact capture time not independently established","artifactSha256":"c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d"},{"sourceId":"harmbench-author-source","url":"https://arxiv.org/html/2402.04249v2","locator":"§4 HarmBench","version":"Selected original resource https://arxiv.org/html/2402.04249v2; sha256:c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d; archived source capture; exact capture time not independently established","artifactSha256":"c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d"},{"sourceId":"harmbench-author-source","url":"https://arxiv.org/html/2402.04249v2","locator":"§6.1 Main Results","version":"Selected original resource https://arxiv.org/html/2402.04249v2; sha256:c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d; archived source capture; exact capture time not independently established","artifactSha256":"c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d"},{"sourceId":"harmbench-author-source","url":"https://arxiv.org/html/2402.04249v2","locator":"§7 Conclusion","version":"Selected original resource https://arxiv.org/html/2402.04249v2; sha256:c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d; archived source capture; exact capture time not independently established","artifactSha256":"c0222f5cb89e2f882de4cbb59e4d61897049528425e5e0cce1928bfb0655e37d"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"bf122be5d7c08f57aae01985345a555eb5e962bed028ef4e597936145b54818d"}},"sources":[{"id":"harmbench","title":"HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal","url":"https://arxiv.org/abs/2402.04249v2","version":"arXiv:2402.04249v2","publicationDate":"2024-02-27","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-24","documentId":"arxiv:2402.04249","authors":["Mantas Mazeika","Long Phan","Xuwang Yin","Andy Zou","Zifan Wang","Norman Mu","Elham Sakhaee","Nathaniel Li","Steven Basart","Bo Li","David Forsyth","Dan Hendrycks"],"aliases":["https://doi.org/10.48550/arXiv.2402.04249","https://arxiv.org/abs/2402.04249","https://arxiv.org/pdf/2402.04249"]}]}