NUWA Risk Evaluation · Method
Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceEvidence updateInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
Adaptive stopping must preserve rare-event sensitivity and state its uncertainty convention.
Research question
When can adaptive stopping save evaluation budget without obscuring rare outcomes?
Design and assumptions
The method stops sampling according to posterior precision and reports savings across nine validation cells. The stopping rule uses a 97% credible interval; other figures and cells use separately stated uncertainty conventions.
Source findings
Planned trial reductions ranged from 57.2% to 97.3% across the nine cells. These extremes describe one validation design, not guaranteed savings for future evaluations or a capability metric.
NUWA interpretation
Meeting an interval-width target does not guarantee coverage or preserve sensitivity to rare dangerous behaviour. Savings need interpretation with the stopping rule and low-probability performance.
Limits and contrary evidence
The main validation configuration is limited. Different interval conventions cannot be silently relabeled as 95% confidence intervals.
Research and decision relevance
NUWA recommendation: Use the existing optstop record to compare planned and realised sampling, with rare-event diagnostics and explicit interval type.
Precise source locations
- Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations · §§2–4; Fig.2–3; Appendix B; text lines 00186,00772Original resource captured 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Evidence update · 2026-10-28
Model-assisted reading · independent evidence and bilingual editorial review pending. Values are source findings or inventory counts, separate from the trend result series.
The stopping rule uses a 97% credible interval, not a generic 95% confidence interval. Savings of 57.2%–97.3% describe nine validation cells; reaching a width target does not guarantee coverage or rare-behaviour sensitivity.
Source-bound observations (2)
Counts, rates and contrasts have different units. A sample count is not a number of independent experiments. Unknown numerators and interval levels remain unknown.
| System / metric | Reported value | Population and conditions |
|---|---|---|
| Planned trial reduction | 57.2 % | Conditions, limitations and source{
"precisionThreshold": 0.05,
"stoppingCredibleLevel": 0.97,
"validation": "nine cells;200items except mid-binary198;10epochs"
}Extremal cells across one validation design; not guaranteed future savings or a model capability score. Original source ↗ · §3.1; text lines 00188,00707Original capture 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137 |
| Planned trial reduction | 97.3 % | Conditions, limitations and source{
"precisionThreshold": 0.05,
"stoppingCredibleLevel": 0.97,
"validation": "nine cells;200items except mid-binary198;10epochs"
}Extremal cells across one validation design; not guaranteed future savings or a model capability score. Original source ↗ · §3.1; text lines 00188,00959Original capture 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137 |
Sources for this interpretation
- Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations · §§2–4; Fig.2–3; Appendix B; text lines 00186,00772Original resource captured 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137
Institutions and roles
Authors (1)
Toby D. Pilditch
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{optstop,
title = {Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations},
url = {https://www.aisi.gov.uk/research/knowing-when-to-stop-bayesian-optimal-stopping-for-llm-evaluations},
author = {Toby D. Pilditch},
year = {2026},
date = {2026-08-14},
version = {Source capture 2026-10-25},
note = {Source capture 2026-10-25}
}
NUWA record · BibTeX
@misc{optstop-nuwa,
title = {Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/optstop/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--optstop.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--optstop.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] Toby D. Pilditch. Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations. 2026-08-14
- [2] Toby D. Pilditch. Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations. 2026-08-14 ; Author affiliations on the original paper title page; §§2–4; Fig.2–3; Appendix B; text lines 00186,00772