NUWA Risk Evaluation · Method
HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceEvidence updateInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
Nested evaluation structure belongs in the uncertainty model.
Research question
How should nested evaluation units enter an uncertainty model?
Design and assumptions
The official methods article uses hierarchical Bayesian modelling to represent nested variation across items, repeated runs and raters, including partial pooling. This reading covers the article, not a complete paper and code audit.
Source findings
Nested models can distinguish sources of evaluation variability that a simple pooled mean hides, while introducing assumptions about hierarchy, priors and pooling.
NUWA interpretation
Uncertainty should follow sampling and dependence structure. A more complex model is not automatically more reliable; priors, diagnostics and misspecification still matter.
Limits and contrary evidence
Bayesian credible intervals are not automatically frequentist confidence intervals. Hierarchical assumptions and partial pooling require validation.
Research and decision relevance
NUWA recommendation: Register task, run and rater units with interval kind and method before aggregating repeated evaluation results.
Precise source locations
- HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling · Official methods article; text lines 00001,00012,00013,00018,00021,00024,00025,00027Original resource captured 2026-10-28T07:24:46.595313+00:00; sha256:c82cd12ddaeb77a8e3cb6508891e42849838f2e9b1485f4a61b71f4eac5547b0
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
Institutions and roles
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{hibayes,
title = {HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling},
url = {https://www.aisi.gov.uk/blog/hibayes-improving-llm-evaluation-with-hierarchical-bayesian-modelling},
version = {Original resource captured 2026-10-28T07:24:46.595313+00:00; sha256:c82cd12ddaeb77a8e3cb6508891e42849838f2e9b1485f4a61b71f4eac5547b0},
note = {Original resource captured 2026-10-28T07:24:46.595313+00:00; sha256:c82cd12ddaeb77a8e3cb6508891e42849838f2e9b1485f4a61b71f4eac5547b0}
}
NUWA record · BibTeX
@misc{hibayes-nuwa,
title = {HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/hibayes/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--hibayes.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--hibayes.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] HiBayES: Improving LLM evaluation with hierarchical Bayesian modelling ; Official methods article; text lines 00001,00012,00013,00018,00021,00024,00025,00027