Risk Evaluation · Method

Guidelines for capability elicitation

NUWA research note · independent review pending

First published: 2024-03-15Source checked: 2026-10-25

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: low observed performance is evidence of inability only after justified elicitation and validity checks.

Research question

How can evaluators avoid mistaking scaffold or task-access failures for model inability?

Design and assumptions

METR’s guidelines separate basic elicitation, failure handling, red-flag checks, human baselining and reporting, while distinguishing spurious from real failures.

Source findings

The document targets capabilities accessible with plausible post-training enhancement, not just default chat performance.

It also includes checks against overestimation such as memorization.

NUWA interpretation

NUWA interpretation: low observed performance is evidence of inability only after justified elicitation and validity checks.

Limits and contrary evidence

These are guidelines designed around general autonomy, not an empirical guarantee that every capability has been elicited.

Research and decision relevance

Model recommendation: record failure taxonomy, elicitation changes and baseline conditions in the evaluation report.

Precise source locations

Guidelines for capability elicitation · §1 OverviewSelected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established
Guidelines for capability elicitation · §2.1 Basic elicitationSelected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established
Guidelines for capability elicitation · §2.2 Handle remaining failuresSelected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established
Guidelines for capability elicitation · §3 Categorizing failure typesSelected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established
Guidelines for capability elicitation · §5 Recommendations for avoiding overestimation of capabilitiesSelected original resource https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/; sha256:6203c5d9af6dafc58c41650e57a00a542ec581d04d6cd584c71c6a8076ca823a; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Published by METR

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{capability-elicitation,
  title = {Guidelines for capability elicitation},
  url = {https://metr.org/blog/2024-03-15-guidelines-for-capability-elicitation/},
  year = {2024},
  date = {2024-03-15},
  version = {Source capture 2026-10-25},
  note = {Source capture 2026-10-25}
}

NUWA record · BibTeX

@misc{capability-elicitation-nuwa,
  title = {Guidelines for capability elicitation — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/capability-elicitation/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--capability-elicitation.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--capability-elicitation.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] Guidelines for capability elicitation. 2024-03-15 ; Official publication masthead and article byline; §1 Overview; §2.1 Basic elicitation