NUWA RESEARCH RECORD
TASTE: Can AI Models Judge AI Safety Research Proposals?
原始文献 ↗ · BibTeX · RIS原文报告Fable 5总准确率60%、Kimi K3为59%，估计人类一致率为77%。展示研究问题时，Fable 5在74组不同问题来源的提案对中为69%，在18组同一问题的提案对中为31%。总成绩掩盖了明显的任务子集差异。
研究问题
模型能否判断安全研究提案的价值？
设计与前提
TASTE 包含50份实证安全研究提案形成的92组偏好对，标签筛选讨论后强置信判断及至少两分的评分差。Table 3 对提案顺序和问题可见条件的偏好概率先平均，再给出每对判断；Table 4–6 分别拆解留出研究者分歧、问题类型敏感性和标签筛选。各模型API与推理预算不同。
研究发现
原文报告Fable 5总准确率60%、Kimi K3为59%，估计人类一致率为77%。展示研究问题时，Fable 5在74组不同问题来源的提案对中为69%，在18组同一问题的提案对中为31%。总成绩掩盖了明显的任务子集差异。
局限与反证
标签测量与特定研究者的偏好一致性，不能验证研究的实际价值。92组比较复用提案，频次上限不能消除依赖。Fable 5部分拒答采用原文规定的替代规则，其中包括其他模型回答。舍入百分比不能反推精确分子。其余数字图形和附录正文尚未完成穷尽提取。
来源证据
| 系统 | 测量 | 原值 | 条件与位置 |
|---|---|---|---|
| — | fable5-pairwise-agreement | 60%percentn = 92 preference-pairs{"text":"Per-model confidence intervals span roughly ±10 percentage points","type":"approximately-described-confidence-interval","confidenceLevel":null,"lower":null,"upper":null,"locator":"Evaluating Models’ Research Judgment, final figure caption","scope":"General caption across evaluated models; not a precisely reported Fable 5 interval."} | Standard pairwise setup; human preference labels; not factual accuracy.Evaluating Models’ Research Judgment |
| — | estimated-human-agreement | 77%percentn = 92 preference-pairs | Opposing-pair sampled researcher scores, not conventional same-rater agreement.Building a Research Judgment Benchmark (TASTE) |
| Fable 5 | proposal-preference-accuracy | 60%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Fable 5 |
| Kimi K3 | proposal-preference-accuracy | 59%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Kimi K3 |
| Grok 4.5 | proposal-preference-accuracy | 58%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Grok 4.5 |
| Gemini 3.5 Flash | proposal-preference-accuracy | 56%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Gemini 3.5 Flash |
| Sonnet 4.6 | proposal-preference-accuracy | 56%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Sonnet 4.6 |
| Haiku 4.5 | proposal-preference-accuracy | 56%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Haiku 4.5 |
| Opus 4.8 | proposal-preference-accuracy | 55%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.8 |
| Opus 4.5 | proposal-preference-accuracy | 54%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.5 |
| Sonnet 5 | proposal-preference-accuracy | 52%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Sonnet 5 |
| Qwen3.7-Max | proposal-preference-accuracy | 52%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Qwen3.7-Max |
| Opus 5 | proposal-preference-accuracy | 51%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 5 |
| Muse Spark 1.1 | proposal-preference-accuracy | 51%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Muse Spark 1.1 |
| Opus 4.7 | proposal-preference-accuracy | 49%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.7 |
| GPT-5.6-luna | proposal-preference-accuracy | 48%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GPT-5.6-luna |
| GPT-5.6-terra | proposal-preference-accuracy | 48%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GPT-5.6-terra |
| GPT-5.6-sol | proposal-preference-accuracy | 48%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GPT-5.6-sol |
| GLM 5.2 | proposal-preference-accuracy | 46%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GLM 5.2 |
| Opus 4.6 | proposal-preference-accuracy | 45%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.6 |
| Gemini 3.1 Pro | proposal-preference-accuracy | 41%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Gemini 3.1 Pro |
| Fable 5 | proposal-preference-accuracy | 69%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, distinct-prompt, question shown |
| Fable 5 | proposal-preference-accuracy | 61%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, distinct-prompt, question hidden |
| Fable 5 | proposal-preference-accuracy | 31%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, within-prompt, question shown |
| Fable 5 | proposal-preference-accuracy | 47%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, within-prompt, question hidden |
| Kimi K3 | proposal-preference-accuracy | 66%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, distinct-prompt, question shown |
| Kimi K3 | proposal-preference-accuracy | 65%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, distinct-prompt, question hidden |
| Kimi K3 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, within-prompt, question shown |
| Kimi K3 | proposal-preference-accuracy | 53%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, within-prompt, question hidden |
| Grok 4.5 | proposal-preference-accuracy | 57%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, distinct-prompt, question shown |
| Grok 4.5 | proposal-preference-accuracy | 56%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, distinct-prompt, question hidden |
| Grok 4.5 | proposal-preference-accuracy | 28%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, within-prompt, question shown |
| Grok 4.5 | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, within-prompt, question hidden |
| Gemini 3.5 Flash | proposal-preference-accuracy | 44%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, distinct-prompt, question shown |
| Gemini 3.5 Flash | proposal-preference-accuracy | 63%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, distinct-prompt, question hidden |
| Gemini 3.5 Flash | proposal-preference-accuracy | 56%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, within-prompt, question shown |
| Gemini 3.5 Flash | proposal-preference-accuracy | 67%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, within-prompt, question hidden |
| Sonnet 4.6 | proposal-preference-accuracy | 57%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, distinct-prompt, question shown |
| Sonnet 4.6 | proposal-preference-accuracy | 60%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, distinct-prompt, question hidden |
| Sonnet 4.6 | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, within-prompt, question shown |
| Sonnet 4.6 | proposal-preference-accuracy | 44%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, within-prompt, question hidden |
| Haiku 4.5 | proposal-preference-accuracy | 60%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, distinct-prompt, question shown |
| Haiku 4.5 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, distinct-prompt, question hidden |
| Haiku 4.5 | proposal-preference-accuracy | 67%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, within-prompt, question shown |
| Haiku 4.5 | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, within-prompt, question hidden |
| Opus 4.8 | proposal-preference-accuracy | 59%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, distinct-prompt, question shown |
| Opus 4.8 | proposal-preference-accuracy | 61%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, distinct-prompt, question hidden |
| Opus 4.8 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, within-prompt, question shown |
| Opus 4.8 | proposal-preference-accuracy | 44%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, within-prompt, question hidden |
| Opus 4.5 | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, distinct-prompt, question shown |
| Opus 4.5 | proposal-preference-accuracy | 50%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, distinct-prompt, question hidden |
| Opus 4.5 | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, within-prompt, question shown |
| Opus 4.5 | proposal-preference-accuracy | 47%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, within-prompt, question hidden |
| Sonnet 5 | proposal-preference-accuracy | 54%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, distinct-prompt, question shown |
| Sonnet 5 | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, distinct-prompt, question hidden |
| Sonnet 5 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, within-prompt, question shown |
| Sonnet 5 | proposal-preference-accuracy | 42%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, within-prompt, question hidden |
| Qwen3.7-Max | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, distinct-prompt, question shown |
| Qwen3.7-Max | proposal-preference-accuracy | 55%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, distinct-prompt, question hidden |
| Qwen3.7-Max | proposal-preference-accuracy | 36%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, within-prompt, question shown |
| Qwen3.7-Max | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, within-prompt, question hidden |
| Opus 5 | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, distinct-prompt, question shown |
| Opus 5 | proposal-preference-accuracy | 59%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, distinct-prompt, question hidden |
| Opus 5 | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, within-prompt, question shown |
| Opus 5 | proposal-preference-accuracy | 39%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, within-prompt, question hidden |
| Muse Spark 1.1 | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, distinct-prompt, question shown |
| Muse Spark 1.1 | proposal-preference-accuracy | 54%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, distinct-prompt, question hidden |
| Muse Spark 1.1 | proposal-preference-accuracy | 35%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, within-prompt, question shown |
| Muse Spark 1.1 | proposal-preference-accuracy | 56%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, within-prompt, question hidden |
| Opus 4.7 | proposal-preference-accuracy | 55%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, distinct-prompt, question shown |
| Opus 4.7 | proposal-preference-accuracy | 55%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, distinct-prompt, question hidden |
| Opus 4.7 | proposal-preference-accuracy | 25%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, within-prompt, question shown |
| Opus 4.7 | proposal-preference-accuracy | 36%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, within-prompt, question hidden |
| GPT-5.6-luna | proposal-preference-accuracy | 54%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, distinct-prompt, question shown |
| GPT-5.6-luna | proposal-preference-accuracy | 47%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, distinct-prompt, question hidden |
| GPT-5.6-luna | proposal-preference-accuracy | 28%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, within-prompt, question shown |
| GPT-5.6-luna | proposal-preference-accuracy | 53%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, within-prompt, question hidden |
| GPT-5.6-terra | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, distinct-prompt, question shown |
| GPT-5.6-terra | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, distinct-prompt, question hidden |
| GPT-5.6-terra | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, within-prompt, question shown |
| GPT-5.6-terra | proposal-preference-accuracy | 44%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, within-prompt, question hidden |
| GPT-5.6-sol | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, distinct-prompt, question shown |
| GPT-5.6-sol | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, distinct-prompt, question hidden |
| GPT-5.6-sol | proposal-preference-accuracy | 28%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, within-prompt, question shown |
| GPT-5.6-sol | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, within-prompt, question hidden |
| GLM 5.2 | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, distinct-prompt, question shown |
| GLM 5.2 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, distinct-prompt, question hidden |
| GLM 5.2 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, within-prompt, question shown |
| GLM 5.2 | proposal-preference-accuracy | 39%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, within-prompt, question hidden |
| Opus 4.6 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, distinct-prompt, question shown |
| Opus 4.6 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, distinct-prompt, question hidden |
| Opus 4.6 | proposal-preference-accuracy | 31%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, within-prompt, question shown |
| Opus 4.6 | proposal-preference-accuracy | 31%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, within-prompt, question hidden |
| Gemini 3.1 Pro | proposal-preference-accuracy | 47%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, distinct-prompt, question shown |
| Gemini 3.1 Pro | proposal-preference-accuracy | 43%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, distinct-prompt, question hidden |
| Gemini 3.1 Pro | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, within-prompt, question shown |
| Gemini 3.1 Pro | proposal-preference-accuracy | 56%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, within-prompt, question hidden |
| Fable 5 | proposal-preference-accuracy | 64%%n = 63 preference-pairs | dyad agrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, dyad agrees |
| Fable 5 | proposal-preference-accuracy | 56%%n = 16 preference-pairs | dyad disagrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, dyad disagrees |
| Fable 5 | proposal-preference-accuracy | 50%%n = 6 preference-pairs | dyad split; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, dyad split |
| Fable 5 | proposal-preference-accuracy | 43%%n = 7 preference-pairs | no held-out vote; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, no held-out vote |
| Kimi K3 | proposal-preference-accuracy | 57%%n = 63 preference-pairs | dyad agrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, dyad agrees |
| Kimi K3 | proposal-preference-accuracy | 56%%n = 16 preference-pairs | dyad disagrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, dyad disagrees |
| Kimi K3 | proposal-preference-accuracy | 83%%n = 6 preference-pairs | dyad split; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, dyad split |
| Kimi K3 | proposal-preference-accuracy | 57%%n = 7 preference-pairs | no held-out vote; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, no held-out vote |
| R0 | human-preference-agreement | 58%%n = 12 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R0, all solo |
| R1 | human-preference-agreement | 59%%n = 38 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R1, all solo |
| R1 | human-preference-agreement | 62%%n = 30 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R1, strong post-discussion |
| R1 | human-agreement-change | +2percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R1, delta |
| R2 | human-preference-agreement | 46%%n = 36 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R2, all solo |
| R2 | human-preference-agreement | 62%%n = 8 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R2, strong post-discussion |
| R2 | human-agreement-change | +17percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R2, delta |
| R3 | human-preference-agreement | 60%%n = 53 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R3, all solo |
| R3 | human-preference-agreement | 100%%n = 7 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R3, strong post-discussion |
| R3 | human-agreement-change | +40percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R3, delta |
| R4 | human-preference-agreement | 53%%n = 37 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R4, all solo |
| R4 | human-preference-agreement | 67%%n = 3 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R4, strong post-discussion |
| R4 | human-agreement-change | +14percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R4, delta |
| R5 | human-preference-agreement | 46%%n = 26 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R5, all solo |
| R5 | human-preference-agreement | 75%%n = 4 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R5, strong post-discussion |
| R5 | human-agreement-change | +29percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R5, delta |
| R6 | human-preference-agreement | 47%%n = 36 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R6, all solo |
| R7 | human-preference-agreement | 45%%n = 37 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R7, all solo |
| R7 | human-preference-agreement | 50%%n = 2 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R7, strong post-discussion |
| R7 | human-agreement-change | +5percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R7, delta |
| R8 | human-preference-agreement | 61%%n = 23 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R8, all solo |
| R8 | human-preference-agreement | 75%%n = 4 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R8, strong post-discussion |
| R8 | human-agreement-change | +14percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R8, delta |
| R9 | human-preference-agreement | 60%%n = 30 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R9, all solo |
阅读范围与复核
Selected original webpage sections; linked paper, appendix and data not fully audited.
模型辅助整理，独立证据复核待完成。
核对原始来源 ↗