NUWA Evaluation dossier
BioTIER
Evaluates refusal of high-risk requests alongside preservation of benign biological assistance.
Task and measurement
Benchmark protocol profile
- Task / subject
- Models responding to tiered biological-risk prompts
- Measurement target
- Response and refusal behavior across expert-curated prompt sets.
- Interpretation boundary
- Refusal is a safeguard outcome; it does not directly measure underlying biological knowledge or task ability.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can biological safeguards distinguish narrowly high-risk requests from benign scientific access?
- Task population
- 542 expert-curated prompts span three risk sets; 52 models from ten developers are assessed with binary substantive-response/refusal labels.
- Scoring and failure modes
- Binary substantive-answer/refusal compliance, with restricted-refuse and benign-permit tasks separated.
- System conditions
- 542 prompts, 52 models; highest publicly available reasoning configurations.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Binary compliance does not score answer correctness, completeness or real-world uplift (§3.3).
- Validity and interpretation
- Taxonomy thresholds involve expert judgment and need periodic updating; benchmark-aware training remains possible (§5.4).
- Appropriate research use
- Model recommendation: report under-refusal and over-refusal separately and avoid interpreting compliance as an operational-harm score.
- BioTIER: A Refusal Benchmark for Targeted Biological Risk Mitigation · AbstractSelected original resource https://arxiv.org/html/2607.14479v1; sha256:d39bab620389ee6f797ce66762dc0bda4f5afdf89d5653bcbd08a2f0970b9e01; archived source capture; exact capture time not independently established
- BioTIER: A Refusal Benchmark for Targeted Biological Risk Mitigation · §3.3 EvaluationSelected original resource https://arxiv.org/html/2607.14479v1; sha256:d39bab620389ee6f797ce66762dc0bda4f5afdf89d5653bcbd08a2f0970b9e01; archived source capture; exact capture time not independently established
- BioTIER: A Refusal Benchmark for Targeted Biological Risk Mitigation · §5.4 LimitationsSelected original resource https://arxiv.org/html/2607.14479v1; sha256:d39bab620389ee6f797ce66762dc0bda4f5afdf89d5653bcbd08a2f0970b9e01; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
CBRN / Hazardous knowledge →
Response handling supplies safeguard context for this risk topic, not a hazardous-capability score.
References
- [1] Eleanor M. Marshall, Pedro Medeiros, et al.. BioTIER: A Refusal Benchmark for Targeted Biological Risk Mitigation. 2026-07-16. v1 [arXiv]; Author affiliations on the original paper title page; Figure 6