NUWA Risk Evaluation · Method
Introducing Bloom : an open source tool for automated behavioral evaluations
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: the seed is part of the measurement definition, not an optional implementation detail.
Research question
Can generated evaluations measure researcher-specified behaviors reproducibly?
Design and assumptions
Bloom generates scenarios from behavior descriptions and example transcripts; an evaluation seed records configuration, and generated outcomes are judged automatically.
Source findings
The release demonstrates four behaviors across sixteen models and compares judgments with hand labels.
Absolute metrics changed with configuration, while a self-preferential-bias ranking was largely stable.
NUWA interpretation
NUWA interpretation: the seed is part of the measurement definition, not an optional implementation detail.
Limits and contrary evidence
Dynamic scenario generation and filtering affect the tested population; validation on selected behaviors does not establish universal judge accuracy.
Research and decision relevance
Model recommendation: cite the seed, generation/filtering settings and validation task with every result.
Precise source locations
- Introducing Bloom : an open source tool for automated behavioral evaluations · How Bloom worksSelected original resource https://www.anthropic.com/research/bloom; sha256:67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de; archived source capture; exact capture time not independently established
- Introducing Bloom : an open source tool for automated behavioral evaluations · Validation and trustSelected original resource https://www.anthropic.com/research/bloom; sha256:67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de; archived source capture; exact capture time not independently established
- Introducing Bloom : an open source tool for automated behavioral evaluations · Case study: Self-preferential biasSelected original resource https://www.anthropic.com/research/bloom; sha256:67c6404128375c3d8baa90e80084fed90b800eb20485afb1d51468ea0a6ca1de; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{bloom,
title = {Introducing Bloom : an open source tool for automated behavioral evaluations},
url = {https://www.anthropic.com/research/bloom},
year = {2025},
date = {2025-12-19},
version = {Source capture 2026-10-25},
note = {Source capture 2026-10-25}
}
NUWA record · BibTeX
@misc{bloom-nuwa,
title = {Introducing Bloom : an open source tool for automated behavioral evaluations — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/bloom/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--bloom.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--bloom.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] Introducing Bloom : an open source tool for automated behavioral evaluations. 2025-12-19 ; Official publication masthead and article byline; How Bloom works; Validation and trust