Evaluation dossier

Virology Capabilities Test (VCT)

Measures knowledge and troubleshooting responses in specialist virology questions.

SecureBio · Federal University of ABC · Center for AI Safety · Massachusetts Institute of Technology / arXiv:2504.16137v2 / Capability

Sources and provenance ↓ Read the research narrative →

Task and measurement

Benchmark protocol profile

Task / subject: Models answering specialist virology troubleshooting questions
Measurement target: Question-level performance in a versioned multimodal test.
Scoring / outcome: Multiple-response accuracy · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability.
Interpretation boundary: Knowledge-question accuracy does not establish practical harmful-task execution; v1 and v2 scores are not directly interchangeable.

A reported evaluation

The source evaluates Claude 3.5 Sonnet (Jun ’24) · Table 1 · Virology Capabilities Test (VCT).

Reported measurement: Multiple-response accuracy · 26.9%.

Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. Three evaluation runs; source reports SEM below 1% for all models.

Source-reported measurements for Claude 3.5 Sonnet (Jun ’24) in Table 1; values are transcribed, not independently reproduced.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: How well do models answer expert-authored virology troubleshooting questions?
Task population: 322 multimodal items use all-or-nothing multiple-response scoring; experts are compared within their areas of specialization.
Scoring and failure modes: All-or-nothing selection of all correct statements; Hamming mismatch is a separate analysis.
System conditions: 322 multimodal questions; human expertise matched within subareas.
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: o3 scored 43.8%, versus a 22.1% mean for expert virologists with internet access. The benchmark also analyzes response mismatches and alternative answer formats (§5.1).
Validity and interpretation: Experts may disagree about answer keys; embodied and communal tacit knowledge are incompletely represented (§6; Appendix A3).
Appropriate research use: Model recommendation: retain expertise matching and scoring format; use this only as nonoperational capability evidence.

Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark · AbstractSelected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established
Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark · §5.1 SetupSelected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established
Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark · §6 DiscussionSelected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established
Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark · Appendix A3 Limitations of This WorkSelected original resource https://arxiv.org/html/2504.16137v2; sha256:3b0bc61ab556258ca26318138a0ac00dbfb59bb8d3318a8038b71a6788be0d99; archived source capture; exact capture time not independently established

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark

Actual evaluators: Not disclosed

12 result records · 12 reported system names

Protocols and comparison decisions (0)

Original source charts and materials

Reported results

Source order · reported configurations

Multiple-response accuracy

0–100%

Claude 3.5 Sonnet (Jun ’24)

Claude 3.5 Sonnet (Jun ’24) · 26.9% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

26.9%

Claude 3.5 Sonnet (Oct ’24)

Claude 3.5 Sonnet (Oct ’24) · 33.6% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

33.6%

Claude 3.7 Sonnet (Feb ’25)*

Claude 3.7 Sonnet (Feb ’25)* · 30.8% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

30.8%

Gemini 1.5 Pro

Gemini 1.5 Pro · 25.4% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

25.4%

Gemini 2.0 Pro (exp.)

Gemini 2.0 Pro (exp.) · 27.4% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

27.4%

Gemini 2.5 Pro (exp.)

Gemini 2.5 Pro (exp.) · 37.6% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · One evaluation run (italicized in the source table).

37.6%

GPT-4o (Nov ’24)

GPT-4o (Nov ’24) · 18.8% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

18.8%

GPT-4.1 (Apr ’25)

GPT-4.1 (Apr ’25) · 26.6% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

26.6%

GPT-4.5 Preview (Feb ’25)

GPT-4.5 Preview (Feb ’25) · 28.3% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

28.3%

o1 (Dec ’24)

o1 (Dec ’24) · 35.4% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · One evaluation run (italicized in the source table).

35.4%

o3 (Apr ’25)

o3 (Apr ’25) · 43.8% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · Three evaluation runs; source reports SEM below 1% for all models.

43.8%

o4-mini (Apr ’25)

o4-mini (Apr ’25) · 37.0% · Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. · One evaluation run (italicized in the source table).

37.0%

Showing 12 of 12 systems

Values describe the measured outcome under the source protocol, not a general safety score.

Table 1: multiple-response accuracy. Three evaluation runs, except italicized source entries evaluated once; the asterisk denotes up to 4096 reasoning tokens. Knowledge accuracy is not end-to-end harmful capability. Evaluation date not specified; publication / record dates are shown below.

Evaluation conditions and system configurations

Claude 3.5 Sonnet (Jun ’24): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Claude 3.5 Sonnet (Oct ’24): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Claude 3.7 Sonnet (Feb ’25)*: Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Gemini 1.5 Pro: Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Gemini 2.0 Pro (exp.): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Gemini 2.5 Pro (exp.): One evaluation run (italicized in the source table). · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

GPT-4o (Nov ’24): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

GPT-4.1 (Apr ’25): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

GPT-4.5 Preview (Feb ’25): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

o1 (Dec ’24): One evaluation run (italicized in the source table). · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

o3 (Apr ’25): Three evaluation runs; source reports SEM below 1% for all models. · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

o4-mini (Apr ’25): One evaluation run (italicized in the source table). · Table 1 · Virology Capabilities Test (VCT); modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified

Reported results · source order
System / configuration	Multiple-response accuracy	Publication / verification
Claude 3.5 Sonnet (Jun ’24)Harness not specified	26.9%	Source checked2026-10-25
Claude 3.5 Sonnet (Oct ’24)Harness not specified	33.6%	Source checked2026-10-25
Claude 3.7 Sonnet (Feb ’25)*Harness not specified	30.8%	Source checked2026-10-25
Gemini 1.5 ProHarness not specified	25.4%	Source checked2026-10-25
Gemini 2.0 Pro (exp.)Harness not specified	27.4%	Source checked2026-10-25
Gemini 2.5 Pro (exp.)Harness not specified	37.6%	Source checked2026-10-25
GPT-4o (Nov ’24)Harness not specified	18.8%	Source checked2026-10-25
GPT-4.1 (Apr ’25)Harness not specified	26.6%	Source checked2026-10-25
GPT-4.5 Preview (Feb ’25)Harness not specified	28.3%	Source checked2026-10-25
o1 (Dec ’24)Harness not specified	35.4%	Source checked2026-10-25
o3 (Apr ’25)Harness not specified	43.8%	Source checked2026-10-25
o4-mini (Apr ’25)Harness not specified	37.0%	Source checked2026-10-25

References

Numerical results are reported by the cited authors.

[1] Jasper Götting a, Pedro Medeiros a, et al.. Virology Capabilities Test (VCT): A Multimodal Virology Q&A Benchmark. 2025-04-29. v2 [arXiv]; Author affiliations on the original paper title page; Table 1

Search NUWA Index

Virology Capabilities Test (VCT)

Task and measurement

A reported evaluation

Evidence across reports