CharXiv

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

How well do multimodal models understand naturally occurring scientific charts?

Task population

2,323 expert-curated arXiv charts support over 10,000 descriptive/reasoning questions, split into 1,000 validation charts and a held-answer test set.

Scoring and failure modes

Separate descriptive/reasoning accuracy; validation and held-answer test populations differ.

System conditions

2,323 charts; 1,000 validation charts and remainder for test.

Statistical meaning

Source-reported result statements; this does not establish trial counts, denominators or an estimator: The paper reports GPT-4o reasoning accuracy of 47.1%. Human reasoning/descriptive accuracy was 80.5%/92.1%, and the authors flag grading or participant error as possible contributors.

Validity and interpretation

Human errors and automated grading uncertainty limit ceiling claims; chart curation is expensive and scientific-paper coverage is selective.

Appropriate research use

Model recommendation: distinguish descriptive from reasoning questions and validation from held-out test results.

• CharXiv · AbstractSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
• CharXiv · §3 CharXiv: A Real-World and Challenging Chart Understanding BenchmarkSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
• CharXiv · §4.2 Experimental ResultsSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established
• CharXiv · §5 Conclusion: LimitationsSelected original resource https://arxiv.org/html/2406.18521; sha256:5d25e15e0ce1e63c2a5dca2d34c10b62b82cb45702f4071b8e870bd903cc5d30; archived source capture; exact capture time not independently established