NUWA Risk Evaluation · Study
Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch connectionsResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: multiple judges do not guarantee independent or valid measurement.
Research question
Can a panel of LLM judges still systematically misgrade safeguard responses?
Design and assumptions
Generality audits FORTRESS’s Inspect dataset, solver, scorer and multi-judge voting, separating rubric fulfillment from actual model-response content.
Source findings
Regrading reduced reported ARS by up to 35% and ORS by up to 88%.
Judges sometimes scored the adversarial request rather than an empty response; voting rules amplified the error.
NUWA interpretation
NUWA interpretation: multiple judges do not guarantee independent or valid measurement.
Limits and contrary evidence
The audit concerns a particular implementation and regrading policy; its score reductions are not newly measured changes in model safeguards.
Research connections
This article audits FORTRESS’s scoring implementation; it is a methodological follow-up, not an independent benchmark population.
Research and decision relevance
Model recommendation: test empty responses, abstentions and voting logic before using composite judge-panel scores.
Precise source locations
- Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness · Eval architecture overviewSelected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established
- Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness · FORTRESS’s Fragile Judging PanelSelected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established
- Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness · What does FORTRESS actually measure?Selected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established
- Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness · Constructing stronger FORTRESSesSelected original resource https://generality.org/blog/posts/fortress-fragile-judges/; sha256:ac0c18b1ee9c3d14472e55bf6a660ab6f53f21cba2eab8ef6b7e254139c97a4d; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Authors (1)
Laurence Wroe
Related work
FORTRESS: Frontier Risk Evaluation for National Security and Public Safety
Audits how the FORTRESS grading procedure affects reported scores.
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{fortress-audit,
title = {Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness},
url = {https://generality.org/blog/posts/fortress-fragile-judges/},
author = {Laurence Wroe},
version = {Source capture 2026-10-25},
note = {Source capture 2026-10-25}
}
NUWA record · BibTeX
@misc{fortress-audit-nuwa,
title = {Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/fortress-audit/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--fortress-audit.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--fortress-audit.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] Laurence Wroe. Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness ; Article byline, 18 September 2026; What is FORTRESS?; audit methodology; Research background and discussion in the original publication; Eval architecture overview