{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"fortress-audit","title":"Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness","aliases":[],"authors":["Laurence Wroe"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"study","sourceId":"fortress-audit-publication","attributionSourceId":"fortress-audit-publication","originalUrl":"https://generality.org/blog/posts/fortress-fragile-judges/","publicationDate":null,"updatedOn":"2026-10-27","areas":["cbrn","cybersecurity"],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"generality","role":"research","sourceId":"fortress-audit-publication","locator":"Article byline, 18 September 2026; What is FORTRESS?; audit methodology"},{"organizationId":"generality","role":"publisher","sourceId":"fortress-audit-publication","locator":"Article byline, 18 September 2026; What is FORTRESS?; audit methodology"}],"summary":{"en":"Audits grading failures in FORTRESS and examines how judge errors alter risk and over-refusal scores.","zh":"审计 FORTRESS 的评分失败，研究评判错误如何影响风险和过度拒答分数。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":["evaluation-audit"]},"reading":{"id":"fortress-audit","title":"Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness","en":{"question":"Can a panel of LLM judges still systematically misgrade safeguard responses?","design":"Generality audits FORTRESS’s Inspect dataset, solver, scorer and multi-judge voting, separating rubric fulfillment from actual model-response content.","findings":["Regrading reduced reported ARS by up to 35% and ORS by up to 88%.","Judges sometimes scored the adversarial request rather than an empty response; voting rules amplified the error."],"interpretation":"NUWA interpretation: multiple judges do not guarantee independent or valid measurement.","limits":"The audit concerns a particular implementation and regrading policy; its score reductions are not newly measured changes in model safeguards.","decisionRelevance":"Model recommendation: test empty responses, abstentions and voting logic before using composite judge-panel scores.","takeaway":"NUWA interpretation: multiple judges do not guarantee independent or valid measurement.","connections":["This article audits FORTRESS’s scoring implementation; it is a methodological follow-up, not an independent benchmark population."],"scoring":"Regraded ARS and ORS changes under audited judge/abstention/voting behavior.","conditions":null,"statistics":"Regrading reduced reported ARS by up to 35% and ORS by up to 88%.","usage":"Model recommendation: test empty responses, abstentions and voting logic before using composite judge-panel scores."},"zh":{"question":"多个模型判官仍会系统性误判防护回答吗？","design":"Generality审计FORTRESS的Inspect数据、求解器、计分和投票，区分量表满足与真实回答内容。","findings":["重判使ARS最高降35%、ORS最高降88%。","判官有时对攻击请求而非空回答判分，投票规则又放大错误。"],"interpretation":"NUWA 解读：多判官不保证独立或有效测量。","limits":"审计针对特定实现及重判政策，降分不是模型防护本身发生变化。","decisionRelevance":"模型建议：使用多判官综合分前测试空回答、弃权及投票逻辑。","takeaway":"NUWA 解读：多判官不保证独立或有效测量。","connections":["本文审计FORTRESS计分实现，是方法跟进，不是独立基准总体。"],"scoring":"审计判官、弃权及投票规则后重算ARS与ORS变化。","conditions":null,"statistics":"重判使ARS最高降35%、ORS最高降88%。","usage":"模型建议：使用多判官综合分前测试空回答、弃权及投票逻辑。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"fortress-audit-publication","url":"https://generality.org/blog/posts/fortress-fragile-judges/","locator":"Eval architecture overview","version":"Selected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established","artifactSha256":"ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d"},{"sourceId":"fortress-audit-publication","url":"https://generality.org/blog/posts/fortress-fragile-judges/","locator":"FORTRESS’s Fragile Judging Panel","version":"Selected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established","artifactSha256":"ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d"},{"sourceId":"fortress-audit-publication","url":"https://generality.org/blog/posts/fortress-fragile-judges/","locator":"What does FORTRESS actually measure?","version":"Selected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established","artifactSha256":"ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d"},{"sourceId":"fortress-audit-publication","url":"https://generality.org/blog/posts/fortress-fragile-judges/","locator":"Constructing stronger FORTRESSes","version":"Selected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established","artifactSha256":"ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"e4762a7e65e5258f9548985ee537ad5a30f00e3bbde76a869bc5e0cd9c94b6bf"}},"sources":[{"id":"fortress-audit-publication","title":"Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness","url":"https://generality.org/blog/posts/fortress-fragile-judges/","version":"Source capture 2026-10-25","publicationDate":null,"datePrecision":"unknown","type":"research-publication","verifiedOn":"2026-10-27","documentId":"https://generality.org/blog/posts/fortress-fragile-judges","authors":["Laurence Wroe"]}]}