NUWA RESEARCH RECORD
Measuring what Matters: Construct Validity in Large Language Model Benchmarks
Original publication ↗ · BibTeX · RISThe review identifies recurring gaps between constructs, operational tasks and resulting claims, and proposes a practical checklist for benchmark design and interpretation.
Research question
Do benchmark tasks and scores support the phenomena that their authors claim to measure?
Design & assumptions
A systematic review uses an expert codebook to examine phenomenon definitions, task sampling, scoring and validity claims in benchmark papers.
Findings
The review identifies recurring gaps between constructs, operational tasks and resulting claims, and proposes a practical checklist for benchmark design and interpretation.
Limits & counterevidence
The conference and keyword selection excludes parts of the literature. Checklist coverage is not proof that a benchmark is valid, and missed reporting does not establish that a procedure was never performed.
Source evidence
No structured numerical extraction is available in this reading package.
Reading scope & review
Selected original HTML sections and table context; not a complete paper/appendix audit or independent reproduction.
Model-assisted reading; independent evidence review pending.
Check the original source ↗