{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"vct","title":"Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark","aliases":["Virology Capabilities Test (VCT)"],"authors":["Jasper Götting","Pedro Medeiros","Jon G Sanders","Nathaniel Li","Long Phan","Karam Elabd","Lennart Justen","Dan Hendrycks","Seth Donoughe"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"vct","attributionSourceId":"vct-author-source","originalUrl":"https://arxiv.org/abs/2504.16137v2","publicationDate":"2025-04-21","updatedOn":"2026-10-25","areas":["cbrn"],"scopeNote":{"en":"Primary dimension is question-answering capability; version and answer-key validity remain material.","zh":"主维度是问答能力；版本与答案效度是重要限制。"},"protocolIds":["vct"],"contributions":[{"organizationId":"securebio","role":"research","sourceId":"vct-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"ufabc","role":"research","sourceId":"vct-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"cais","role":"research","sourceId":"vct-author-source","locator":"Author affiliations on the original paper title page"},{"organizationId":"mit","role":"research","sourceId":"vct-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Measures knowledge and troubleshooting responses in specialist virology questions.","zh":"测量专业病毒学问题中的知识与排障响应。"},"note":{"question":{"en":"Question-level performance in a versioned multimodal test.","zh":"有版本的多模态题目测试表现。"},"method":{"en":"Models answering specialist virology troubleshooting questions","zh":"回答专门病毒学排障问题的模型"},"finding":{"en":"Measures knowledge and troubleshooting responses in specialist virology questions.","zh":"测量专业病毒学问题中的知识与排障响应。"},"supports":{"en":"Primary dimension is question-answering capability; version and answer-key validity remain material.","zh":"主维度是问答能力；版本与答案效度是重要限制。"},"limits":{"en":"Knowledge-question accuracy does not establish practical harmful-task execution; v1 and v2 scores are not directly interchangeable.","zh":"知识问答准确率不证明实际有害任务执行能力；v1 与 v2 分数不可直接互换。"},"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"vct","title":"Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark","en":{"question":"How well do models answer expert-authored virology troubleshooting questions?","design":"322 multimodal items use all-or-nothing multiple-response scoring; experts are compared within their areas of specialization.","findings":["o3 scored 43.8%, versus a 22.1% mean for expert virologists with internet access.","The benchmark also analyzes response mismatches and alternative answer formats (§5.1)."],"interpretation":"NUWA interpretation: expert-question performance is not demonstrated wet-lab user uplift.","limits":"Experts may disagree about answer keys; embodied and communal tacit knowledge are incompletely represented (§6; Appendix A3).","decisionRelevance":"Model recommendation: retain expertise matching and scoring format; use this only as nonoperational capability evidence.","takeaway":"NUWA interpretation: expert-question performance is not demonstrated wet-lab user uplift.","connections":[],"scoring":"All-or-nothing selection of all correct statements; Hamming mismatch is a separate analysis.","conditions":"322 multimodal questions; human expertise matched within subareas.","statistics":"o3 scored 43.8%, versus a 22.1% mean for expert virologists with internet access. The benchmark also analyzes response mismatches and alternative answer formats (§5.1).","usage":"Model recommendation: retain expertise matching and scoring format; use this only as nonoperational capability evidence."},"zh":{"question":"模型回答专家编写的病毒学问题表现如何？","design":"322道多模态题采用多选全对计分，人类比较限于各自专业范围。","findings":["o3为43.8%，可联网专家均值22.1%。","论文另分析答案集合差异和替代格式（§5.1）。"],"interpretation":"NUWA 解读：专家题表现不是已证明的湿实验用户增益。","limits":"专家可能对答案有分歧，具身与协作隐性知识覆盖有限（§6、附录A3）。","decisionRelevance":"模型建议：保留专业匹配和计分格式，仅作为非操作性能力证据。","takeaway":"NUWA 解读：专家题表现不是已证明的湿实验用户增益。","connections":[],"scoring":"全部正确陈述全选才计分，汉明差异另分析。","conditions":"322多模态题，人类在各自子专业内比较。","statistics":"o3为43.8%，可联网专家均值22.1%。 论文另分析答案集合差异和替代格式（§5.1）。","usage":"模型建议：保留专业匹配和计分格式，仅作为非操作性能力证据。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"vct-author-source","url":"https://arxiv.org/html/2504.16137v2","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established","artifactSha256":"3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99"},{"sourceId":"vct-author-source","url":"https://arxiv.org/html/2504.16137v2","locator":"§5.1 Setup","version":"Selected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established","artifactSha256":"3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99"},{"sourceId":"vct-author-source","url":"https://arxiv.org/html/2504.16137v2","locator":"§6 Discussion","version":"Selected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established","artifactSha256":"3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99"},{"sourceId":"vct-author-source","url":"https://arxiv.org/html/2504.16137v2","locator":"Appendix A3 Limitations of This Work","version":"Selected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established","artifactSha256":"3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"70894ebbc01badc9aa30ad17747c8f9bf87430f820c4dd621aac9a54dcc4745f"}},"sources":[{"id":"vct","title":"Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark","url":"https://arxiv.org/abs/2504.16137v2","version":"arXiv:2504.16137v2","publicationDate":"2025-04-29","datePrecision":"day","type":"benchmark","verifiedOn":"2026-10-25","documentId":"arxiv:2504.16137","authors":["Jasper Götting a","Pedro Medeiros a","Jon G Sanders a","Nathaniel Li †","Long Phan","Karam Elabd","Lennart Justen","Dan Hendrycks","Seth Donoughe ∗"],"aliases":["https://doi.org/10.48550/arXiv.2504.16137","https://arxiv.org/abs/2504.16137","https://arxiv.org/pdf/2504.16137"]}]}