{"schemaVersion":"nuwa-record-evidence-1","batchId":"research-2026-10-28","baseSnapshotId":"2026-10-27.3","publication":"published","dossier":{"id":"item-response-safety","en":{"takeaway":"Headline cohort size differs from complete-case and latent-analysis denominators.","question":"Do safety benchmarks measure common latent traits across the same model population?","design":"The study uses item response theory to analyse latent structure across safety benchmarks, with different complete-case populations at each stage: up to 192 headline models, 169 in the complete benchmark pool and 134 in latent analysis. Items are filtered from 5,255 to 5,067.","findings":["The analysis reveals patterns of shared item response, but factor labels summarise loadings rather than independently validated real-risk constructs. Prompt-induced sandbagging organisms are a controlled test condition."],"interpretation":"Latent variables can expose item redundancy, difficulty and response structure, without becoming a universal risk score. Filtering and scoring failures change the population to which the results apply.","limits":"Calibration distribution shift, judgment failures and prompt-induced organisms limit extrapolation to concealed strategic behaviour. Do not use 192 as the latent-analysis denominator.","connections":[],"decisionRelevance":"NUWA recommendation: Update population counts and construct-validity notes on the existing record; inspect held-out calibration before using IRT for allocation decisions."},"zh":{"takeaway":"总体至多 192 模型，但完整跨基准池 169，潜变量结构分析只有 134；项目由 5,255 变为过滤后 5,067。","question":"安全分数是否由相同潜在特征驱动？","design":"研究用项目反应理论分析跨安全基准的潜在结构，并比较不同阶段的完整样本。总体至多 192 个模型，完整跨基准池 169 个，潜变量结构分析为 134 个；项目从 5,255 个过滤至 5,067 个。","findings":["研究揭示项目反应的共同模式，但因子名称是载荷概括，而非独立验证的现实风险构念。提示式藏拙模型属于受控试验条件。"],"interpretation":"潜变量可以帮助检查题目冗余、难度与模型表现结构，却不能自动成为统一风险分数。样本筛选与判分失败会改变可解释总体。","limits":"因子名称只是载荷模式总结；校准分布变化、判分失败与提示式藏拙模型限制外推。","connections":[],"decisionRelevance":"NUWA 建议：补充既有条目的条件分母；不把 IRT 变成全球统一风险评分。"},"assertionPolicy":{"findings":"source-finding","interpretation":"nuwa-interpretation","decisionRelevance":"recommendation"},"observations":[{"id":"irt-cohort","sourceId":"dr-20261028-irt-safety","metric":"Models in headline cohort","value":192,"unit":"models","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","locator":"Methods; Table1; text lines 00071,00088,00101,00142,00192,00460","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026","capturedAt":"2026-10-28T07:25:52.921099+00:00","title":"Item Response Theory for AI Safety","version":"Original capture 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"limitations":"Different analysis populations; do not use192as factor-analysis denominator."},{"id":"irt-complete","sourceId":"dr-20261028-irt-safety","metric":"Complete benchmark models","value":169,"unit":"models","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","locator":"Methods; Table1; text lines 00142,00238,00462","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026","capturedAt":"2026-10-28T07:25:52.921099+00:00","title":"Item Response Theory for AI Safety","version":"Original capture 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"limitations":"Different analysis populations; do not use192as factor-analysis denominator."},{"id":"irt-latent","sourceId":"dr-20261028-irt-safety","metric":"Latent-analysis models","value":134,"unit":"models","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","locator":"Methods; Table1; text lines 00142,00196","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026","capturedAt":"2026-10-28T07:25:52.921099+00:00","title":"Item Response Theory for AI Safety","version":"Original capture 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"limitations":"Different analysis populations; do not use192as factor-analysis denominator."},{"id":"irt-items-raw","sourceId":"dr-20261028-irt-safety","metric":"Initial item inventory","value":5255,"unit":"items","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","locator":"Methods; Table1; text lines 00088,00103,00162","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026","capturedAt":"2026-10-28T07:25:52.921099+00:00","title":"Item Response Theory for AI Safety","version":"Original capture 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"limitations":"Different analysis populations; do not use192as factor-analysis denominator."},{"id":"irt-items-retained","sourceId":"dr-20261028-irt-safety","metric":"Retained items","value":5067,"unit":"items","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","locator":"Methods; Table1; text lines 00140,00145","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026","capturedAt":"2026-10-28T07:25:52.921099+00:00","title":"Item Response Theory for AI Safety","version":"Original capture 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"limitations":"Different analysis populations; do not use192as factor-analysis denominator."}],"relatedWorkIds":[],"supplementSourceId":"irt-safety","batchId":"research-2026-10-28","sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","title":"Item Response Theory for AI Safety","locator":"Methods; Table1; results; limitations; text lines 00142,00196","version":"Original resource captured 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","capturedAt":"2026-10-28T07:25:52.921099+00:00","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-original-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"3eb5dd3aa8a289370b9e7d63105cbabb386bf5de9fcadd4f216ff868b7292e15"}},"update":{"id":"item-response-safety","review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-original-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"3eb5dd3aa8a289370b9e7d63105cbabb386bf5de9fcadd4f216ff868b7292e15"},"text":{"en":"The original-source supplement distinguishes the headline 192 models, 169 complete benchmark models and 134 latent-analysis models, with filtering from 5,255 to 5,067 items. Factor names summarise loadings rather than validated universal risk constructs.","zh":"本轮原文补充区分 192 个总体模型、169 个完整跨基准模型和 134 个潜变量分析模型，保留 5,255 至 5,067 的项目筛选。因子名称概括载荷，不能当作已验证的统一风险构念。"},"sourceRefs":[{"sourceId":"dr-20261028-irt-safety","url":"https://arxiv.org/html/2608.05086","title":"Item Response Theory for AI Safety","locator":"Methods; Table1; results; limitations; text lines 00142,00196","version":"Original resource captured 2026-10-28T07:25:52.921099+00:00; sha256:b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","artifactSha256":"b73cdc52885df6e9c0e26f0fcc68480d3e3222ed79c4b6c6106ec2e00203b492","textSha256":"2a8f5a1004a01b04d7cb957d02cb1f6d2a2bea917edc13f90aa67e29ffac65de","capturedAt":"2026-10-28T07:25:52.921099+00:00","resourceEdition":"arXiv:2608.05086v1 [cs.AI] 05 Aug 2026"}],"relatedWorkIds":[],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"}}}