NUWA RESEARCH RECORD
evalci: A Python Library for Statistically Rigorous Comparison of Language Model Evaluations
Original publication ↗ · BibTeX · RISIt makes statistical comparisons easier to compute, but its illustrative leaderboard case reconstructs item vectors from aggregate accuracies rather than using original paired logs.
Research question
How can evaluation tables retain uncertainty and paired-comparison semantics?
Design & assumptions
The library packages interval estimation, paired tests, clustering, multiplicity and power calculations, with reference checks for implementations.
Findings
It makes statistical comparisons easier to compute, but its illustrative leaderboard case reconstructs item vectors from aggregate accuracies rather than using original paired logs.
Limits & counterevidence
That reconstruction cannot recover actual item correlations or repair mismatched evaluation protocols. Do not import the example significance claims into NUWA as independently re-analyzed observed comparisons.
Source evidence
No structured numerical extraction is available in this reading package.
Reading scope & review
Selected original HTML sections and table context; not a complete paper/appendix audit or independent reproduction.
Model-assisted reading; independent evidence review pending.
Check the original source ↗