{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"bloom","title":"Introducing Bloom : an open source tool for automated behavioral evaluations","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"method","sourceId":"bloom-publication","attributionSourceId":"bloom-publication","originalUrl":"https://www.anthropic.com/research/bloom","publicationDate":"2025-12-19","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"anthropic","role":"publisher","sourceId":"bloom-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Generates behavioral evaluations from specified target behaviors and examines variation across frontier models.","zh":"从指定目标行为生成行为评测，并研究不同前沿模型的表现差异。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"bloom","title":"Introducing Bloom : an open source tool for automated behavioral evaluations","en":{"question":"Can generated evaluations measure researcher-specified behaviors reproducibly?","design":"Bloom generates scenarios from behavior descriptions and example transcripts; an evaluation seed records configuration, and generated outcomes are judged automatically.","findings":["The release demonstrates four behaviors across sixteen models and compares judgments with hand labels.","Absolute metrics changed with configuration, while a self-preferential-bias ranking was largely stable."],"interpretation":"NUWA interpretation: the seed is part of the measurement definition, not an optional implementation detail.","limits":"Dynamic scenario generation and filtering affect the tested population; validation on selected behaviors does not establish universal judge accuracy.","decisionRelevance":"Model recommendation: cite the seed, generation/filtering settings and validation task with every result.","takeaway":"NUWA interpretation: the seed is part of the measurement definition, not an optional implementation detail.","connections":[],"scoring":"Generated-scenario behavior frequency and severity, always tied to the evaluation seed.","conditions":null,"statistics":null,"usage":"Model recommendation: cite the seed, generation/filtering settings and validation task with every result."},"zh":{"question":"生成式评测能否可复现地测研究者指定行为？","design":"Bloom依据行为描述和例轨生成场景，以评测种子记录配置，自动判断结果。","findings":["发布展示16模型的四类行为，并与人工标签核对。","绝对指标随配置改变，一个自偏好研究的排序较稳定。"],"interpretation":"NUWA 解读：种子属于测量定义，不是可省略实现细节。","limits":"动态生成及筛选改变受测总体，特定行为验证不证明普遍判官准确。","decisionRelevance":"模型建议：每项结果同时引用种子、生成筛选设置及验证任务。","takeaway":"NUWA 解读：种子属于测量定义，不是可省略实现细节。","connections":[],"scoring":"生成场景中的行为频率及严重性，始终绑定评测种子。","conditions":null,"statistics":"发布展示16模型的四类行为，并与人工标签核对。","usage":"模型建议：每项结果同时引用种子、生成筛选设置及验证任务。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"bloom-publication","url":"https://www.anthropic.com/research/bloom","locator":"How Bloom works","version":"Selected original resource https://www.anthropic.com/research/bloom; sha256:67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de; archived source capture; exact capture time not independently established","artifactSha256":"67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de"},{"sourceId":"bloom-publication","url":"https://www.anthropic.com/research/bloom","locator":"Validation and trust","version":"Selected original resource https://www.anthropic.com/research/bloom; sha256:67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de; archived source capture; exact capture time not independently established","artifactSha256":"67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de"},{"sourceId":"bloom-publication","url":"https://www.anthropic.com/research/bloom","locator":"Case study: Self-preferential bias","version":"Selected original resource https://www.anthropic.com/research/bloom; sha256:67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de; archived source capture; exact capture time not independently established","artifactSha256":"67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"08e0967a4d367cf39cf2b90739dd70c572f49ad2e41569d82540a83f5f24a8f2"}},"sources":[{"id":"bloom-publication","title":"Introducing Bloom : an open source tool for automated behavioral evaluations","url":"https://www.anthropic.com/research/bloom","version":"Source capture 2026-10-25","publicationDate":"2025-12-19","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://www.anthropic.com/research/bloom"}]}