{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"charxiv","title":"CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs","aliases":["CharXiv"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"charxiv-primary","attributionSourceId":"charxiv-primary","originalUrl":"https://charxiv.github.io/","publicationDate":null,"updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["charxiv"],"contributions":[],"summary":{"en":"Can models interpret research charts?","zh":"模型能否理解研究图表？"},"note":{"question":{"en":"Can models interpret research charts?","zh":"模型能否理解研究图表？"},"method":{"en":"Tests understanding and reasoning over charts. Neo’s withheld-image adaptation measures attempts without the required input.","zh":"检验图表理解与推理。Neo 移除图像的改造测量缺少必要输入时的回答尝试。"},"finding":null,"supports":null,"limits":{"en":"The original visual capability benchmark and the missing-image behavioral test measure different constructs.","zh":"原始视觉能力基准与缺失图像行为测试测量不同构念。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"charxiv","title":"CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs","en":{"question":"How well do multimodal models understand naturally occurring scientific charts?","design":"2,323 expert-curated arXiv charts support over 10,000 descriptive/reasoning questions, split into 1,000 validation charts and a held-answer test set.","findings":["The paper reports GPT-4o reasoning accuracy of 47.1%.","Human reasoning/descriptive accuracy was 80.5%/92.1%, and the authors flag grading or participant error as possible contributors."],"interpretation":"NUWA interpretation: realistic chart composition exposes gaps hidden by simplified chart benchmarks.","limits":"Human errors and automated grading uncertainty limit ceiling claims; chart curation is expensive and scientific-paper coverage is selective.","decisionRelevance":"Model recommendation: distinguish descriptive from reasoning questions and validation from held-out test results.","takeaway":"NUWA interpretation: realistic chart composition exposes gaps hidden by simplified chart benchmarks.","connections":[],"scoring":"Separate descriptive/reasoning accuracy; validation and held-answer test populations differ.","conditions":"2,323 charts; 1,000 validation charts and remainder for test.","statistics":"The paper reports GPT-4o reasoning accuracy of 47.1%. Human reasoning/descriptive accuracy was 80.5%/92.1%, and the authors flag grading or participant error as possible contributors.","usage":"Model recommendation: distinguish descriptive from reasoning questions and validation from held-out test results."},"zh":{"question":"多模态模型能否理解自然科研图表？","design":"2,323张专家精选arXiv图表含逾一万描述推理题，其中1,000图作验证，其余测试答案保留。","findings":["论文报告GPT-4o推理正确率47.1%。","人类推理、描述为80.5%、92.1%，作者指出判分或参与者错误可能影响。"],"interpretation":"NUWA 解读：真实图表组合揭示简化基准隐藏的能力差距。","limits":"人类错误及自动判分不确定限制上限结论，专家构造昂贵且覆盖经选择。","decisionRelevance":"模型建议：区分描述与推理，以及验证与保留测试结果。","takeaway":"NUWA 解读：真实图表组合揭示简化基准隐藏的能力差距。","connections":[],"scoring":"分描述与推理正确率，验证与保留答案测试总体不同。","conditions":"2,323图，1,000验证，其余测试。","statistics":"论文报告GPT-4o推理正确率47.1%。 人类推理、描述为80.5%、92.1%，作者指出判分或参与者错误可能影响。","usage":"模型建议：区分描述与推理，以及验证与保留测试结果。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"charxiv-primary","url":"https://arxiv.org/html/2406.18521","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established","artifactSha256":"5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30"},{"sourceId":"charxiv-primary","url":"https://arxiv.org/html/2406.18521","locator":"§3 CharXiv: A Real-World and Challenging Chart Understanding Benchmark","version":"Selected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established","artifactSha256":"5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30"},{"sourceId":"charxiv-primary","url":"https://arxiv.org/html/2406.18521","locator":"§4.2 Experimental Results","version":"Selected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established","artifactSha256":"5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30"},{"sourceId":"charxiv-primary","url":"https://arxiv.org/html/2406.18521","locator":"§5 Conclusion: Limitations","version":"Selected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established","artifactSha256":"5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"f0ae67a81d99b254c15cef0e0f3491240da488284687e40dab87e43e4638cba0"}},"sources":[{"id":"charxiv-primary","title":"CharXiv","url":"https://charxiv.github.io/","version":"2026-10-25 source capture","publicationDate":null,"datePrecision":"unknown","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"https://charxiv.github.io"}]}