{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"optstop","title":"Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations","aliases":[],"authors":["Toby D. Pilditch"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"method","sourceId":"optstop-publication","attributionSourceId":"optstop-publication-author-source","originalUrl":"https://www.aisi.gov.uk/research/knowing-when-to-stop-bayesian-optimal-stopping-for-llm-evaluations","publicationDate":"2026-08-14","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"aisi","role":"research","sourceId":"optstop-publication-author-source","locator":"Author affiliations on the original paper title page"}],"summary":{"en":"Uses Bayesian optimal stopping to decide when evaluation sampling provides enough information for a stated decision.","zh":"使用贝叶斯最优停止方法，判断评测抽样何时已为指定决策提供足够信息。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[{"label":"Read the full paper","url":"https://arxiv.org/abs/2608.14425"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"optstop","en":{"takeaway":"Adaptive stopping must preserve rare-event sensitivity and state its uncertainty convention.","question":"When can adaptive stopping save evaluation budget without obscuring rare outcomes?","design":"The method stops sampling according to posterior precision and reports savings across nine validation cells. The stopping rule uses a 97% credible interval; other figures and cells use separately stated uncertainty conventions.","findings":["Planned trial reductions ranged from 57.2% to 97.3% across the nine cells. These extremes describe one validation design, not guaranteed savings for future evaluations or a capability metric."],"interpretation":"Meeting an interval-width target does not guarantee coverage or preserve sensitivity to rare dangerous behaviour. Savings need interpretation with the stopping rule and low-probability performance.","limits":"The main validation configuration is limited. Different interval conventions cannot be silently relabeled as 95% confidence intervals.","connections":[],"decisionRelevance":"NUWA recommendation: Use the existing optstop record to compare planned and realised sampling, with rare-event diagnostics and explicit interval type."},"zh":{"takeaway":"九种验证设置的节省为约 57%–97%；停止规则用 97% 可信区间，另有不同统计区间，不能统一为 95% CI。","question":"何时停止采样，既节省预算又保持精度？","design":"方法按后验精度决定何时停止采样，九个验证单元报告相对计划试验量的节省。停止规则使用 97% 可信区间；文中其他图表和单元的区间约定分别保留。","findings":["九个验证单元相对计划试验量的节省为 57.2%–97.3%。范围描述一套验证设计，不保证未来评测同样节省，也不是模型能力指标。"],"interpretation":"达到区间宽度目标不等于保证覆盖率，也不自动保持对稀有危险行为的检出能力。资源节省必须连同停止规则和低概率区域表现一起解释。","limits":"单一主要验证配置；区间宽度保证不等于覆盖率保证，极端/稀有结果尤需谨慎。","connections":[],"decisionRelevance":"NUWA 建议：补充既有 optstop 条目的停止条件、反例与节省适用范围。"},"assertionPolicy":{"findings":"source-finding","interpretation":"nuwa-interpretation","decisionRelevance":"recommendation"},"observations":[{"id":"optstop-min","sourceId":"dr-20261028-optimal-stopping","metric":"Planned trial reduction","value":57.2,"unit":"%","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-optimal-stopping","url":"https://arxiv.org/html/2608.14425","artifactSha256":"43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137","textSha256":"d45226d941cdc95750fdb9b831c9ae6b1078310f00e41b771497fa61c382d11f","locator":"§3.1; text lines 00188,00707","resourceEdition":"arXiv:2608.14425v1 [cs.AI] 14 Aug 2026","capturedAt":"2026-10-28T07:25:52.921145+00:00","title":"Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations","version":"Original capture 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"configuration":{"precisionThreshold":0.05,"stoppingCredibleLevel":0.97,"validation":"nine cells;200items except mid-binary198;10epochs"},"limitations":"Extremal cells across one validation design; not guaranteed future savings or a model capability score."},{"id":"optstop-max","sourceId":"dr-20261028-optimal-stopping","metric":"Planned trial reduction","value":97.3,"unit":"%","assertionKind":"source-finding","sourceRefs":[{"sourceId":"dr-20261028-optimal-stopping","url":"https://arxiv.org/html/2608.14425","artifactSha256":"43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137","textSha256":"d45226d941cdc95750fdb9b831c9ae6b1078310f00e41b771497fa61c382d11f","locator":"§3.1; text lines 00188,00959","resourceEdition":"arXiv:2608.14425v1 [cs.AI] 14 Aug 2026","capturedAt":"2026-10-28T07:25:52.921145+00:00","title":"Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations","version":"Original capture 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137"}],"sourceReported":true,"publication":"published","humanEvidenceApproval":null,"comparisonPermission":"descriptive-within-source-only","numerator":null,"denominator":null,"interval":null,"modelId":null,"configuration":{"precisionThreshold":0.05,"stoppingCredibleLevel":0.97,"validation":"nine cells;200items except mid-binary198;10epochs"},"limitations":"Extremal cells across one validation design; not guaranteed future savings or a model capability score."}],"relatedWorkIds":[],"supplementSourceId":"optimal-stopping","batchId":"research-2026-10-28","sourceRefs":[{"sourceId":"dr-20261028-optimal-stopping","url":"https://arxiv.org/html/2608.14425","title":"Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations","locator":"§§2–4; Fig.2–3; Appendix B; text lines 00186,00772","version":"Original resource captured 2026-10-28T07:25:52.921145+00:00; sha256:43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137","artifactSha256":"43a40c11b2812c7aa61e8ae9b4cc9f1a5ba6f870af8e92af928cd1960565f137","textSha256":"d45226d941cdc95750fdb9b831c9ae6b1078310f00e41b771497fa61c382d11f","capturedAt":"2026-10-28T07:25:52.921145+00:00","resourceEdition":"arXiv:2608.14425v1 [cs.AI] 14 Aug 2026"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-original-sections","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"337f081f325c2f0ad1a643a9077f115addb550e855b259eee22e21066194ae85"}},"sources":[{"id":"optstop-publication","title":"Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations","url":"https://www.aisi.gov.uk/research/knowing-when-to-stop-bayesian-optimal-stopping-for-llm-evaluations","version":"Source capture 2026-10-25","publicationDate":"2026-08-14","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://www.aisi.gov.uk/research/knowing-when-to-stop-bayesian-optimal-stopping-for-llm-evaluations","authors":["Toby D. Pilditch"]}]}