NUWA Evaluation dossier
BioMysteryBench
Can agents solve research-oriented data-analysis problems?
Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Assesses scientific analysis over anonymized datasets; versions, human reference subsets and attempt budgets are kept explicit.
- Interpretation boundary
- Aggregate scientific performance does not establish an operational biological misuse pathway.
SaferAI evaluates the public API without developer cooperation, combining its own benchmark runs with explicitly identified external comparison results. The report separates task capability, adversarially elicited behavior and willingness to attempt persuasion; configurations and provenance vary by benchmark.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can bioinformatics agents solve verifiable research questions that differ from exam-style biology?
- Task population
- Tasks have checkable answers but allow varied solution routes; up to five experts attempt each question, and model accuracy averages five trials per problem.
- Scoring and failure modes
- Accuracy averaged over five trials per problem; human-solvable and human-difficult subsets separate.
- System conditions
- Up to five experts per task; 76 human-solvable and 23 retained human-difficult tasks.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: 76 tasks were solved by at least one human. After removing four flawed questions, 23 remained human-difficult; Mythos Preview solved about 30% of that subset.
- Validity and interpretation
- Small selected subsets, expert recruitment and removed flawed questions constrain claims of human superiority; no wet-lab uplift is measured.
- Appropriate research use
- Model recommendation: retain human-solvable/difficult definitions and five-trial averaging; omit operational biological task details.
- BioMysteryBench · Benchmarking models on verifiable biological tasks with BioMysteryBenchSelected original resource https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench; sha256:669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3; archived source capture; exact capture time not independently established
- BioMysteryBench · Human baselining: Human-solvableSelected original resource https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench; sha256:669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3; archived source capture; exact capture time not independently established
- BioMysteryBench · Human baselining: Human-difficultSelected original resource https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench; sha256:669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3; archived source capture; exact capture time not independently established
Original source charts and materials
Protocol scope is catalogued. Exact numerical extraction and an appropriate source figure remain to be reviewed.
Related research questions
CBRN / Task uplift →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] BioMysteryBench
- [2] GLM-5.2 Risk Evaluation Report. 2026-08-02. v1.1 · revised 26 August 2026 ; Report masthead and stated evaluation methodology