NUWA Evaluation dossier
FORTRESS
Evaluates safeguards on national-security and public-safety tasks, separating hazardous requests from benign-use utility.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Evaluates safeguards on national-security and public-safety tasks, separating hazardous requests from benign-use utility.
- Interpretation boundary
- Judge reliability, task definitions and over-refusal affect the meaning of the reported scores.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- How do safeguards balance national/public-safety risk restriction against benign access?
- Task population
- 500 expert-crafted adversarial prompts each have a benign counterpart and a 4–7-item binary rubric, spanning three broad domains.
- Scoring and failure modes
- Average Risk Score and Over-Refusal Score from per-item binary rubrics; not probabilities of realized harm.
- System conditions
- 500 adversarial/benign prompt pairs, single-turn; judge implementation subject to separate audit.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Claude 3.5 Sonnet’s reported ARS/ORS were 14.09/21.80, versus Gemini 2.5 Pro’s 66.29/1.40.
- Validity and interpretation
- Static single-turn prompts, limited domain expertise and sparse niche coverage constrain validity (§6); judge correctness requires separate audit.
- Appropriate research use
- Model recommendation: retain ARS/ORS together and distinguish the original paper’s estimates from subsequent grading audits.
- FORTRESS: Frontier Risk Evaluation for National Security and Public Safety · AbstractSelected original resource https://arxiv.org/html/2506.14922; sha256:6d8ae92cd9c9950fafbc23990e3a18f67cf4f527869648b8351d21e52bc62efc; archived source capture; exact capture time not independently established
- FORTRESS: Frontier Risk Evaluation for National Security and Public Safety · §5.1 Main ResultsSelected original resource https://arxiv.org/html/2506.14922; sha256:6d8ae92cd9c9950fafbc23990e3a18f67cf4f527869648b8351d21e52bc62efc; archived source capture; exact capture time not independently established
- FORTRESS: Frontier Risk Evaluation for National Security and Public Safety · §6 LimitationsSelected original resource https://arxiv.org/html/2506.14922; sha256:6d8ae92cd9c9950fafbc23990e3a18f67cf4f527869648b8351d21e52bc62efc; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] Christina Q. Knight ∗, Kaustubh Deshpande ⋄, et al.. FORTRESS: Frontier Risk Evaluation for National Security and Public Safety. 2025-06-17. v2 [arXiv]; Author affiliations on the original paper title page; Figure 1