{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"safety-benchmark-review","title":"How Should AI Safety Benchmarks Benchmark Safety?","aliases":["safety benchmark review"],"authors":["Cheng Yu","Severin Engelmann","Ruoxuan Cao","Dalia Ali","Orestis Papakyriakopoulos"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"technical-report","sourceId":"dr-20261028-safety-benchmark-review","attributionSourceId":"dr-20261028-safety-benchmark-review","originalUrl":"https://arxiv.org/html/2601.23112","publicationDate":null,"updatedOn":"2026-10-28","areas":[],"scopeNote":{"en":"Selected original sections; local editorial review. Not an independently replicated experiment.","zh":"选读原始材料；本地编辑审阅稿，不是独立复现实验。"},"protocolIds":[],"contributions":[],"summary":{"en":"Construct coverage and deployment calibration must be evaluated separately from leaderboard scores.","zh":"综述编码 210 项评测，讨论构念覆盖、频率与风险概率混淆及代理测量链。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"tags":[],"contentEditionId":"2026-10-28.1"},"reading":{"id":"safety-benchmark-review","en":{"takeaway":"Construct coverage and deployment calibration must be evaluated separately from leaderboard scores.","question":"Which safety claims are actually supported by the design of a benchmark?","design":"The study codes 210 safety benchmarks using its own review framework, examining the links among tasks, proxy measurements, target risks and real-world consequences. This reading covers definitions, findings and limitations; it does not independently repeat the coding.","findings":["The review identifies gaps in construct coverage and in the justification linking benchmark outcomes to safety claims. Its count describes the authors’ reviewed collection, not a quality census of all existing benchmarks."],"interpretation":"Validity review should expose each step of the proxy chain and its missing evidence. Diagnosing measurement weaknesses does not require an unsupported universal quality score.","limits":"Findings depend on inclusion rules and the authors’ coding judgments. This integration is a selected reading, not a reproduction or an exhaustive benchmark review.","connections":[],"decisionRelevance":"NUWA recommendation: Add construct, scoring and external-validity questions to individual protocol dossiers and record where supporting evidence is unavailable."},"zh":{"takeaway":"综述编码 210 项评测，讨论构念覆盖、频率与风险概率混淆及代理测量链。","question":"安全 Benchmark 与现实危害之间相隔哪些代理变量？","design":"研究按自身编码框架审查 210 个安全基准，区分测试任务、代理指标、目标风险与现实后果之间的关系。此次选读定义、结果及局限，而非独立重做所有编码。","findings":["综述指出测量构念覆盖不足，以及基准结果通向安全主张的论证缺口。210 项是作者所审查的集合规模，不是所有基准质量的总体普查。"],"interpretation":"效度审查需要解释每一段代理链及缺失证据。对测量薄弱环节的诊断不应压缩成无依据的统一质量分。","limits":"属于作者的样本编码与方法建议；不是对全球全部基准的客观评级，也不能据此生成 NUWA 总分。","connections":[],"decisionRelevance":"NUWA 建议：将构念、效度、严重性与外推假设作为档案必填审查项。"},"assertionPolicy":{"findings":"source-finding","interpretation":"nuwa-interpretation","decisionRelevance":"recommendation"},"observations":[{"id":"safety-review-count","sourceId":"dr-20261028-safety-benchmark-review","metric":"Reviewed safety benchmarks","value":210,"unit":"benchmarks","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-safety-benchmark-review","url":"https://arxiv.org/html/2601.23112","artifactSha256":"70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114","textSha256":"609f85e4b1458218bc7a1f58bf0223b21b99dc08a7c2194a6c5eb5df1e8dbee9","locator":"Abstract; benchmark design; text lines 00072","resourceEdition":"arXiv:2601.23112v3 [cs.CY] 05 Aug 2026","capturedAt":"2026-10-28T07:24:41.534986+00:00","title":"How Should AI Safety Benchmarks Benchmark Safety?","version":"Original capture 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"limitations":"Inventory count, not outcome denominator or severity."}],"relatedWorkIds":[],"supplementSourceId":"safety-benchmark-review","batchId":"research-2026-10-28","sourceRefs":[{"sourceId":"dr-20261028-safety-benchmark-review","url":"https://arxiv.org/html/2601.23112","title":"How Should AI Safety Benchmarks Benchmark Safety?","locator":"v3 abstract; §§4–7; Appendix D; text lines 00072","version":"Original resource captured 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114","artifactSha256":"70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114","textSha256":"609f85e4b1458218bc7a1f58bf0223b21b99dc08a7c2194a6c5eb5df1e8dbee9","capturedAt":"2026-10-28T07:24:41.534986+00:00","resourceEdition":"arXiv:2601.23112v3 [cs.CY] 05 Aug 2026"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-original-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"839103d9d47ea3aeca6f0ab8a3bcf64c23de10886668ff4bdb33eca53e4aaf6a"}},"sources":[{"id":"dr-20261028-safety-benchmark-review","title":"How Should AI Safety Benchmarks Benchmark Safety?","url":"https://arxiv.org/html/2601.23112","version":"arXiv:2601.23112v3; version header 05 Aug 2026; Original resource captured 2026-10-28T07:24:41.534986+00:00; sha256:70eecb9bfae0a09f68bc913db1ecc53ece14721009038c714976cb3a7f73f114","publicationDate":null,"datePrecision":"unknown","type":"technical-report","verifiedOn":"2026-10-28","documentId":"https://arxiv.org/html/2601.23112","authors":["Cheng Yu","Severin Engelmann","Ruoxuan Cao","Dalia Ali","Orestis Papakyriakopoulos"]}]}