NUWA Evaluation dossier
AbstentionBench
When does a model abstain from questions that cannot be answered reliably?
Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Tests abstention across 20 datasets with unknown, underspecified, false-premise, subjective or outdated questions.
- Interpretation boundary
- Abstention is distinct from a safety refusal and does not alone establish intent or downstream harm.
Tests abstention across 20 datasets with unknown, underspecified, false-premise, subjective or outdated questions.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Do models know when uncertainty or an ill-posed question warrants abstention?
- Task population
- Twenty datasets span unknown answers, underspecification, false premises, subjectivity and outdated information; twenty frontier models are tested with automatic judging.
- Scoring and failure modes
- Answer/abstain judgments over answerable and unanswerable situations.
- System conditions
- Twenty datasets and twenty frontier models; system prompts/reasoning interventions vary.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: The study reports an average 24% degradation in abstention after reasoning fine-tuning.
- Validity and interpretation
- The stated 24% should not be silently relabeled percentage points; dataset/judge choices constrain generalization.
- Appropriate research use
- Model recommendation: retain answerable and unanswerable subsets and evaluate refusal quality separately from correctness.
- AbstentionBench · AbstractSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
- AbstentionBench · §3 AbstentionBenchSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
- AbstentionBench · §4 ExperimentsSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
- AbstentionBench · §5 DiscussionSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
Original source charts and materials
Protocol scope is catalogued. Exact numerical extraction and an appropriate source figure remain to be reviewed.
Related research questions
References
- [1] AbstentionBench. 2025-06-10