NUWA Risk Evaluation · Benchmark
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: realistic chart composition exposes gaps hidden by simplified chart benchmarks.
Research question
How well do multimodal models understand naturally occurring scientific charts?
Design and assumptions
2,323 expert-curated arXiv charts support over 10,000 descriptive/reasoning questions, split into 1,000 validation charts and a held-answer test set.
Source findings
The paper reports GPT-4o reasoning accuracy of 47.1%.
Human reasoning/descriptive accuracy was 80.5%/92.1%, and the authors flag grading or participant error as possible contributors.
NUWA interpretation
NUWA interpretation: realistic chart composition exposes gaps hidden by simplified chart benchmarks.
Limits and contrary evidence
Human errors and automated grading uncertainty limit ceiling claims; chart curation is expensive and scientific-paper coverage is selective.
Research and decision relevance
Model recommendation: distinguish descriptive from reasoning questions and validation from held-out test results.
Precise source locations
- CharXiv · AbstractSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
- CharXiv · §3 CharXiv: A Real-World and Challenging Chart Understanding BenchmarkSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
- CharXiv · §4.2 Experimental ResultsSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
- CharXiv · §5 Conclusion: LimitationsSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{charxiv,
title = {CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs},
url = {https://charxiv.github.io/},
version = {2026-10-25 source capture},
note = {2026-10-25 source capture}
}
NUWA record · BibTeX
@misc{charxiv-nuwa,
title = {CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/charxiv/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--charxiv.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--charxiv.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] CharXiv ; Abstract; §3 CharXiv: A Real-World and Challenging Chart Understanding Benchmark; §4.2 Experimental Results