NUWA Risk Evaluation · Benchmark
AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: improved answer accuracy and improved abstention are different training outcomes.
Research question
Do models know when uncertainty or an ill-posed question warrants abstention?
Design and assumptions
Twenty datasets span unknown answers, underspecification, false premises, subjectivity and outdated information; twenty frontier models are tested with automatic judging.
Source findings
The study reports an average 24% degradation in abstention after reasoning fine-tuning.
Targeted system prompts improve scores but do not solve reliable uncertainty handling.
NUWA interpretation
NUWA interpretation: improved answer accuracy and improved abstention are different training outcomes.
Limits and contrary evidence
The stated 24% should not be silently relabeled percentage points; dataset/judge choices constrain generalization.
Research and decision relevance
Model recommendation: retain answerable and unanswerable subsets and evaluate refusal quality separately from correctness.
Precise source locations
- AbstentionBench · AbstractSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
- AbstentionBench · §3 AbstentionBenchSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
- AbstentionBench · §4 ExperimentsSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
- AbstentionBench · §5 DiscussionSelected original resource https://arxiv.org/html/2506.09038; sha256:3270c69c4a4075490ddef1d8e9a473edb4da508b4dde3b5bcdeeaf4776963942; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{abstentionbench,
title = {AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions},
url = {https://arxiv.org/abs/2506.09038},
year = {2025},
date = {2025-06-10},
version = {2026-10-25 source capture},
note = {2026-10-25 source capture}
}
NUWA record · BibTeX
@misc{abstentionbench-nuwa,
title = {AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/abstentionbench/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--abstentionbench.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--abstentionbench.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] AbstentionBench. 2025-06-10 ; Abstract; §3 AbstentionBench; §4 Experiments