{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"capability-elicitation","title":"Guidelines for capability elicitation","aliases":[],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"method","sourceId":"capability-elicitation-publication","attributionSourceId":"capability-elicitation-publication","originalUrl":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","publicationDate":"2024-03-15","updatedOn":"2026-10-25","areas":[],"scopeNote":{"en":"Relevance is assigned to the research question and stated conditions of the work. A topic tag is not a risk rating.","zh":"风险关联依据研究问题及明确条件建立，主题标签不代表风险评级。"},"protocolIds":[],"contributions":[{"organizationId":"metr","role":"publisher","sourceId":"capability-elicitation-publication","locator":"Official publication masthead and article byline"}],"summary":{"en":"Specifies practices for eliciting model capabilities so evaluation failures are not mistaken for inability.","zh":"提出诱发模型能力的评测实践，避免将评测失败误判为不具备能力。"},"note":{"question":null,"method":null,"finding":null,"supports":null,"limits":null,"connections":null},"resources":[],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"A substantive NUWA research note is not yet available. The repeated draft sections were removed pending review of the original methods, findings and limitations; use the original publication for the research account.","zh":"实质性的 NUWA 研究笔记尚未完成。重复草稿章节已移除，待核查原始方法、发现与限制；研究内容请以原始出版物为准。"},"noteSourceLocator":"No complete section-level method/results/limitations review completed for this note; cached original page and any chart audit retained as discovery evidence.","tags":[]},"reading":{"id":"capability-elicitation","title":"Guidelines for capability elicitation","en":{"question":"How can evaluators avoid mistaking scaffold or task-access failures for model inability?","design":"METR’s guidelines separate basic elicitation, failure handling, red-flag checks, human baselining and reporting, while distinguishing spurious from real failures.","findings":["The document targets capabilities accessible with plausible post-training enhancement, not just default chat performance.","It also includes checks against overestimation such as memorization."],"interpretation":"NUWA interpretation: low observed performance is evidence of inability only after justified elicitation and validity checks.","limits":"These are guidelines designed around general autonomy, not an empirical guarantee that every capability has been elicited.","decisionRelevance":"Model recommendation: record failure taxonomy, elicitation changes and baseline conditions in the evaluation report.","takeaway":"NUWA interpretation: low observed performance is evidence of inability only after justified elicitation and validity checks.","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"Model recommendation: record failure taxonomy, elicitation changes and baseline conditions in the evaluation report."},"zh":{"question":"如何避免把框架或任务访问故障误认作模型无能力？","design":"METR指南分基础激发、失败处理、警讯检查、人类基线和报告，并区分伪失败与真实失败。","findings":["目标是合理后训练增强可访问的能力，而不只默认聊天表现。","同时检查记忆等高估来源。"],"interpretation":"NUWA 解读：低表现只有经合理激发及效度检查后才支持无能力。","limits":"这是围绕通用自主能力设计的指南，不保证所有能力均已激发。","decisionRelevance":"模型建议：在评测报告记录失败分类、激发变更及基线条件。","takeaway":"NUWA 解读：低表现只有经合理激发及效度检查后才支持无能力。","connections":[],"scoring":null,"conditions":null,"statistics":null,"usage":"模型建议：在评测报告记录失败分类、激发变更及基线条件。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"capability-elicitation-publication","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","locator":"§1 Overview","version":"Selected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established","artifactSha256":"6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a"},{"sourceId":"capability-elicitation-publication","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","locator":"§2.1 Basic elicitation","version":"Selected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established","artifactSha256":"6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a"},{"sourceId":"capability-elicitation-publication","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","locator":"§2.2 Handle remaining failures","version":"Selected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established","artifactSha256":"6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a"},{"sourceId":"capability-elicitation-publication","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","locator":"§3 Categorizing failure types","version":"Selected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established","artifactSha256":"6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a"},{"sourceId":"capability-elicitation-publication","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","locator":"§5 Recommendations for avoiding overestimation of capabilities","version":"Selected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established","artifactSha256":"6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"edfca60732357631c0f470908bddd4c8437b18f520debab1a658ada03d76c67b"}},"sources":[{"id":"capability-elicitation-publication","title":"Guidelines for capability elicitation","url":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/","version":"Source capture 2026-10-25","publicationDate":"2024-03-15","datePrecision":"day","type":"research-publication","verifiedOn":"2026-10-25","documentId":"https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation"}]}