NUWA RESEARCH RECORD
AI Evaluation Should Require Standardized Item-Level Data Releases
Original publication ↗ · BibTeX · RISAggregate scores conceal weak items and construct mismatches. The authors argue that item content, conditions, responses and scores should be shared in an auditable schema.
Research question
What evidence must benchmark releases expose for their validity claims to be independently examined?
Design & assumptions
The position paper proposes standardized item-level releases and illustrates item diagnostics and construct analysis through the OpenEval repository.
Findings
Aggregate scores conceal weak items and construct mismatches. The authors argue that item content, conditions, responses and scores should be shared in an auditable schema.
Limits & counterevidence
This is a methodological position with examples, not proof that openness eliminates contamination. The paper explicitly allows controlled access, embargoes and licensing constraints.
Source evidence
No structured numerical extraction is available in this reading package.
Reading scope & review
Selected original HTML sections and table context; not a complete paper/appendix audit or independent reproduction.
Model-assisted reading; independent evidence review pending.
Check the original source ↗