Risk Evaluation · Technical report

How Should AI Safety Benchmarks Benchmark Safety?

NUWA research note · independent review pending

First published: Publication date not verifiedSource checked: 2026-10-28

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Evidence update Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

Construct coverage and deployment calibration must be evaluated separately from leaderboard scores.

Research question

Which safety claims are actually supported by the design of a benchmark?

Design and assumptions

The study codes 210 safety benchmarks using its own review framework, examining the links among tasks, proxy measurements, target risks and real-world consequences. This reading covers definitions, findings and limitations; it does not independently repeat the coding.

Source findings

The review identifies gaps in construct coverage and in the justification linking benchmark outcomes to safety claims. Its count describes the authors’ reviewed collection, not a quality census of all existing benchmarks.

NUWA interpretation

Validity review should expose each step of the proxy chain and its missing evidence. Diagnosing measurement weaknesses does not require an unsupported universal quality score.

Limits and contrary evidence

Findings depend on inclusion rules and the authors’ coding judgments. This integration is a selected reading, not a reproduction or an exhaustive benchmark review.

Research and decision relevance

NUWA recommendation: Add construct, scoring and external-validity questions to individual protocol dossiers and record where supporting evidence is unavailable.

Precise source locations

How Should AI Safety Benchmarks Benchmark Safety? · v3 abstract; §§4–7; Appendix D; text lines 00072Original resource captured 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Evidence update · 2026-10-28

Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.

Source-bound observations (1)

Counts, rates and contrasts have different units. A sample count is not a number of independent experiments. Unknown numerators and interval levels remain unknown.

System / metric	Reported value	Population and conditions
Reviewed safety benchmarks	210 benchmarks	Conditions, limitations and source Inventory count, not outcome denominator or severity. Original source ↗ · Abstract; benchmark design; text lines 00072Original capture 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114

Download source-bound evidence · JSON ↓

Institutions and roles

Affiliation not indexed in this review

Authors (5)

Cheng Yu · Severin Engelmann · Ruoxuan Cao · Dalia Ali · Orestis Papakyriakopoulos

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{safety-benchmark-review,
  title = {How Should AI Safety Benchmarks Benchmark Safety?},
  url = {https://arxiv.org/html/2601.23112},
  author = {Cheng Yu and Severin Engelmann and Ruoxuan Cao and Dalia Ali and Orestis Papakyriakopoulos},
  version = {arXiv:2601.23112v3; version header 05 Aug 2026; Original resource captured 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114},
  note = {arXiv:2601.23112v3; version header 05 Aug 2026; Original resource captured 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114}
}

NUWA record · BibTeX

@misc{safety-benchmark-review-nuwa,
  title = {How Should AI Safety Benchmarks Benchmark Safety? — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/safety-benchmark-review/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--safety-benchmark-review.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--safety-benchmark-review.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Cheng Yu, Severin Engelmann, et al.. How Should AI Safety Benchmarks Benchmark Safety? ; v3 abstract; §§4–7; Appendix D; text lines 00072