WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← Library

Risk Evaluation · Benchmark

FORTRESS: Frontier Risk Evaluation for National Security and Public Safety

NUWA research note · independent review pending

First published: Source checked: 2026-10-25
Original source ↗Cite this record ↓CBRN →Cybersecurity →
NUWA Research NoteResearch questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: restriction and benign usefulness form two axes that a single refusal rate would hide.

Research question

How do safeguards balance national/public-safety risk restriction against benign access?

Design and assumptions

500 expert-crafted adversarial prompts each have a benign counterpart and a 4–7-item binary rubric, spanning three broad domains.

FORTRESS — Figure 1
Figure 1 · Original figure 1. Read alongside the protocol scope and limitations below. Original figure and source ↗

Source findings

Claude 3.5 Sonnet’s reported ARS/ORS were 14.09/21.80, versus Gemini 2.5 Pro’s 66.29/1.40.

These are rubric-derived risk and over-refusal scores, not harm probabilities.

NUWA interpretation

NUWA interpretation: restriction and benign usefulness form two axes that a single refusal rate would hide.

Limits and contrary evidence

Static single-turn prompts, limited domain expertise and sparse niche coverage constrain validity (§6); judge correctness requires separate audit.

Research and decision relevance

Model recommendation: retain ARS/ORS together and distinguish the original paper’s estimates from subsequent grading audits.

Precise source locations

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Research by Scale AI

Authors (7)

Christina Q. Knight · Kaustubh Deshpande · Ved Sirdeshmukh · Meher Mankikar · Scale Red Team · SEAL Research Team · Julian Michael

Evaluation protocols

FORTRESS →

Related work

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSONSuggest a correction
Original publication · BibTeX
@misc{fortress,
  title = {FORTRESS: Frontier Risk Evaluation for National Security and Public Safety},
  url = {https://arxiv.org/abs/2506.14922},
  author = {Christina Q. Knight and Kaustubh Deshpande and Ved Sirdeshmukh and Meher Mankikar and Scale Red Team and SEAL Research Team and Julian Michael},
  year = {2025},
  date = {2025-06-17},
  version = {arXiv:2506.14922v2},
  note = {arXiv:2506.14922v2}
}
NUWA record · BibTeX
@misc{fortress-nuwa,
  title = {FORTRESS: Frontier Risk Evaluation for National Security and Public Safety — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/fortress/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--fortress.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--fortress.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

  1. [1] Christina Q. Knight ∗, Kaustubh Deshpande ⋄, et al.. FORTRESS: Frontier Risk Evaluation for National Security and Public Safety. 2025-06-17. v2 [arXiv]; Author affiliations on the original paper title page; Abstract; §5.1 Main Results
  2. [2] Laurence Wroe. Auditing FORTRESS: A Multi-Judge Panel Doesn’t Guarantee Robustness ; Research background and discussion in the original publication