{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"biomysterybench","title":"Evaluating Claude’s bioinformatics research capabilities with BioMysteryBench","aliases":["BioMysteryBench"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"biomysterybench-primary","attributionSourceId":"biomysterybench-primary","originalUrl":"https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench","publicationDate":null,"updatedOn":"2026-10-25","areas":["cbrn"],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["biomysterybench"],"contributions":[],"summary":{"en":"Can agents solve research-oriented data-analysis problems?","zh":"智能体能否解决面向科研的数据分析问题？"},"note":{"question":{"en":"Can agents solve research-oriented data-analysis problems?","zh":"智能体能否解决面向科研的数据分析问题？"},"method":{"en":"Assesses scientific analysis over anonymized datasets; versions, human reference subsets and attempt budgets are kept explicit.","zh":"评测匿名数据集上的科研分析；明确版本、人类参照子集和尝试预算。"},"finding":null,"supports":null,"limits":{"en":"Aggregate scientific performance does not establish an operational biological misuse pathway.","zh":"汇总科研表现不能证明实际生物滥用路径。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"biomysterybench","title":"Evaluating Claude’s bioinformatics research capabilities with BioMysteryBench","en":{"question":"Can bioinformatics agents solve verifiable research questions that differ from exam-style biology?","design":"Tasks have checkable answers but allow varied solution routes; up to five experts attempt each question, and model accuracy averages five trials per problem.","findings":["76 tasks were solved by at least one human.","After removing four flawed questions, 23 remained human-difficult; Mythos Preview solved about 30% of that subset."],"interpretation":"NUWA interpretation: “human-difficult” means unsolved within this recruited attempt process, not impossible for humans.","limits":"Small selected subsets, expert recruitment and removed flawed questions constrain claims of human superiority; no wet-lab uplift is measured.","decisionRelevance":"Model recommendation: retain human-solvable/difficult definitions and five-trial averaging; omit operational biological task details.","takeaway":"NUWA interpretation: “human-difficult” means unsolved within this recruited attempt process, not impossible for humans.","connections":[],"scoring":"Accuracy averaged over five trials per problem; human-solvable and human-difficult subsets separate.","conditions":"Up to five experts per task; 76 human-solvable and 23 retained human-difficult tasks.","statistics":"76 tasks were solved by at least one human. After removing four flawed questions, 23 remained human-difficult; Mythos Preview solved about 30% of that subset.","usage":"Model recommendation: retain human-solvable/difficult definitions and five-trial averaging; omit operational biological task details."},"zh":{"question":"生信智能体能否解决不同于考试题的可验证研究问题？","design":"任务有可核答案但允许不同路径，每题最多五位专家尝试，模型按每题五次取平均。","findings":["76题至少一位人类解决。","删除四道缺陷题后，23题仍为人类困难，Mythos Preview解其中约30%。"],"interpretation":"NUWA 解读：人类困难只指所招募尝试中未解，并非人类不可能。","limits":"小型筛选集、专家招募和删题限制超人类结论，未测湿实验增益。","decisionRelevance":"模型建议：保留子集定义与五次平均，省略操作性生物题目细节。","takeaway":"NUWA 解读：人类困难只指所招募尝试中未解，并非人类不可能。","connections":[],"scoring":"每题五次平均正确率，分人类可解与困难子集。","conditions":"每题最多五专家，76可解及23保留困难题。","statistics":"76题至少一位人类解决。 删除四道缺陷题后，23题仍为人类困难，Mythos Preview解其中约30%。","usage":"模型建议：保留子集定义与五次平均，省略操作性生物题目细节。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"biomysterybench-primary","url":"https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench","locator":"Benchmarking models on verifiable biological tasks with BioMysteryBench","version":"Selected original resource https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench; sha256:669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3; archived source capture; exact capture time not independently established","artifactSha256":"669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3"},{"sourceId":"biomysterybench-primary","url":"https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench","locator":"Human baselining: Human-solvable","version":"Selected original resource https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench; sha256:669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3; archived source capture; exact capture time not independently established","artifactSha256":"669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3"},{"sourceId":"biomysterybench-primary","url":"https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench","locator":"Human baselining: Human-difficult","version":"Selected original resource https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench; sha256:669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3; archived source capture; exact capture time not independently established","artifactSha256":"669c8f9a60c28899106a0784a0aea320de59a48eed1369395bedd3ec6fa71cc3"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"dfc8735fa53f07c501328c8681f9f51a1580e2c35615c687c1f4ff306a2a0f4b"}},"sources":[{"id":"biomysterybench-primary","title":"BioMysteryBench","url":"https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench","version":"2026-10-25 source capture","publicationDate":null,"datePrecision":"unknown","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"https://www.anthropic.com/research/Evaluating-Claude-For-Bioinformatics-With-BioMysteryBench"}]}