NUWA Evaluation dossier
TruthfulQA
Measures truthful answering on questions designed to elicit common human misconceptions.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Measures truthful answering on questions designed to elicit common human misconceptions.
- Interpretation boundary
- Factual falsehood does not establish intentional deception; retained as foundational measurement context.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can models avoid repeating familiar human misconceptions?
- Task population
- 817 questions across 38 categories target imitative falsehoods, with truthfulness and informativeness evaluated separately.
- Scoring and failure modes
- Truthfulness and informativeness of generated answers, not intention to lie.
- System conditions
- 817 misconception-targeted questions across 38 categories in the original cohort.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: The original study’s best model was truthful on 58% versus a 94% human baseline.
- Validity and interpretation
- Strong performance does not imply truthfulness in specialized domains; historical model-scale findings should not be generalized to all later models.
- Appropriate research use
- Model recommendation: preserve generation cohort and separate truthful uncertainty from informative correctness.
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · AbstractSelected original resource https://arxiv.org/html/2109.07958; sha256:c0181fc915933de5c1d5a93995d17bd531bfbc37639d7dfcf8a5eb6d6fae6f0d; archived source capture; exact capture time not independently established
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · §2 The TruthfulQA BenchmarkSelected original resource https://arxiv.org/html/2109.07958; sha256:c0181fc915933de5c1d5a93995d17bd531bfbc37639d7dfcf8a5eb6d6fae6f0d; archived source capture; exact capture time not independently established
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · §4 ResultsSelected original resource https://arxiv.org/html/2109.07958; sha256:c0181fc915933de5c1d5a93995d17bd531bfbc37639d7dfcf8a5eb6d6fae6f0d; archived source capture; exact capture time not independently established
- TruthfulQA: Measuring How Models Mimic Human Falsehoods · §7 ConclusionSelected original resource https://arxiv.org/html/2109.07958; sha256:c0181fc915933de5c1d5a93995d17bd531bfbc37639d7dfcf8a5eb6d6fae6f0d; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] Stephanie Lin, Jacob Hilton, et al.. TruthfulQA: Measuring how models mimic human falsehoods. 2021-09-08
- [2] Stephanie Lin, Jacob Hilton, et al.. TruthfulQA: Measuring How Models Mimic Human Falsehoods. 2021-09-08 ; Author affiliations on the original paper title page; Figure 2