NUWA RESEARCH RECORD
TASTE: Can AI Models Judge AI Safety Research Proposals?
Original publication ↗ · BibTeX · RISThe source reports headline accuracy of 60% for Fable 5 and 59% for Kimi K3, against an estimated human agreement of 77%. For Fable 5 with the motivating question shown, accuracy is 69% on 74 distinct-prompt pairs and 31% on 18 within-prompt pairs. A headline score hides this substantial task-subset dependence.
Research question
Can models judge promising safety-research proposals?
Design & assumptions
TASTE constructs 92 preference pairs over 50 empirical safety proposals. Labels select strong-confidence, post-discussion judgments with score gaps of at least two points. Table 3 averages model preference probabilities over proposal order and question visibility before assigning a verdict per pair; Tables 4–6 expose held-out disagreement, prompt-type sensitivity and label filtering. Model APIs and reasoning effort differ.
Findings
The source reports headline accuracy of 60% for Fable 5 and 59% for Kimi K3, against an estimated human agreement of 77%. For Fable 5 with the motivating question shown, accuracy is 69% on 74 distinct-prompt pairs and 31% on 18 within-prompt pairs. A headline score hides this substantial task-subset dependence.
Limits & counterevidence
Labels measure agreement with selected researchers, not verified research value. The 92 pairs reuse proposals; the cap limits but does not eliminate dependence. Fable 5 uses a source-described refusal fallback including other model answers. Rounded percentages do not determine exact numerators. Remaining numeric figures and appendix prose are not yet exhaustively extracted.
Source evidence
| System | Measure | Reported value | Conditions & location |
|---|---|---|---|
| — | fable5-pairwise-agreement | 60%percentn = 92 preference-pairs{"text":"Per-model confidence intervals span roughly ±10 percentage points","type":"approximately-described-confidence-interval","confidenceLevel":null,"lower":null,"upper":null,"locator":"Evaluating Models’ Research Judgment, final figure caption","scope":"General caption across evaluated models; not a precisely reported Fable 5 interval."} | Standard pairwise setup; human preference labels; not factual accuracy.Evaluating Models’ Research Judgment |
| — | estimated-human-agreement | 77%percentn = 92 preference-pairs | Opposing-pair sampled researcher scores, not conventional same-rater agreement.Building a Research Judgment Benchmark (TASTE) |
| Fable 5 | proposal-preference-accuracy | 60%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Fable 5 |
| Kimi K3 | proposal-preference-accuracy | 59%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Kimi K3 |
| Grok 4.5 | proposal-preference-accuracy | 58%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Grok 4.5 |
| Gemini 3.5 Flash | proposal-preference-accuracy | 56%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Gemini 3.5 Flash |
| Sonnet 4.6 | proposal-preference-accuracy | 56%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Sonnet 4.6 |
| Haiku 4.5 | proposal-preference-accuracy | 56%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Haiku 4.5 |
| Opus 4.8 | proposal-preference-accuracy | 55%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.8 |
| Opus 4.5 | proposal-preference-accuracy | 54%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.5 |
| Sonnet 5 | proposal-preference-accuracy | 52%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Sonnet 5 |
| Qwen3.7-Max | proposal-preference-accuracy | 52%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Qwen3.7-Max |
| Opus 5 | proposal-preference-accuracy | 51%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 5 |
| Muse Spark 1.1 | proposal-preference-accuracy | 51%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Muse Spark 1.1 |
| Opus 4.7 | proposal-preference-accuracy | 49%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.7 |
| GPT-5.6-luna | proposal-preference-accuracy | 48%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GPT-5.6-luna |
| GPT-5.6-terra | proposal-preference-accuracy | 48%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GPT-5.6-terra |
| GPT-5.6-sol | proposal-preference-accuracy | 48%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GPT-5.6-sol |
| GLM 5.2 | proposal-preference-accuracy | 46%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, GLM 5.2 |
| Opus 4.6 | proposal-preference-accuracy | 45%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Opus 4.6 |
| Gemini 3.1 Pro | proposal-preference-accuracy | 41%%n = 92 preference-pairs | 92 pairs; one verdict per pair after averaging both proposal orderings and question-shown/hidden variants; situational framing. Fable 5 refusal fallback replaces some cells using flipped order, lower effort, then Opus 4.8.PDF p.26, Table 3, Gemini 3.1 Pro |
| Fable 5 | proposal-preference-accuracy | 69%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, distinct-prompt, question shown |
| Fable 5 | proposal-preference-accuracy | 61%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, distinct-prompt, question hidden |
| Fable 5 | proposal-preference-accuracy | 31%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, within-prompt, question shown |
| Fable 5 | proposal-preference-accuracy | 47%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Fable 5, within-prompt, question hidden |
| Kimi K3 | proposal-preference-accuracy | 66%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, distinct-prompt, question shown |
| Kimi K3 | proposal-preference-accuracy | 65%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, distinct-prompt, question hidden |
| Kimi K3 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, within-prompt, question shown |
| Kimi K3 | proposal-preference-accuracy | 53%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Kimi K3, within-prompt, question hidden |
| Grok 4.5 | proposal-preference-accuracy | 57%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, distinct-prompt, question shown |
| Grok 4.5 | proposal-preference-accuracy | 56%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, distinct-prompt, question hidden |
| Grok 4.5 | proposal-preference-accuracy | 28%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, within-prompt, question shown |
| Grok 4.5 | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Grok 4.5, within-prompt, question hidden |
| Gemini 3.5 Flash | proposal-preference-accuracy | 44%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, distinct-prompt, question shown |
| Gemini 3.5 Flash | proposal-preference-accuracy | 63%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, distinct-prompt, question hidden |
| Gemini 3.5 Flash | proposal-preference-accuracy | 56%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, within-prompt, question shown |
| Gemini 3.5 Flash | proposal-preference-accuracy | 67%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.5 Flash, within-prompt, question hidden |
| Sonnet 4.6 | proposal-preference-accuracy | 57%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, distinct-prompt, question shown |
| Sonnet 4.6 | proposal-preference-accuracy | 60%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, distinct-prompt, question hidden |
| Sonnet 4.6 | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, within-prompt, question shown |
| Sonnet 4.6 | proposal-preference-accuracy | 44%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 4.6, within-prompt, question hidden |
| Haiku 4.5 | proposal-preference-accuracy | 60%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, distinct-prompt, question shown |
| Haiku 4.5 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, distinct-prompt, question hidden |
| Haiku 4.5 | proposal-preference-accuracy | 67%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, within-prompt, question shown |
| Haiku 4.5 | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Haiku 4.5, within-prompt, question hidden |
| Opus 4.8 | proposal-preference-accuracy | 59%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, distinct-prompt, question shown |
| Opus 4.8 | proposal-preference-accuracy | 61%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, distinct-prompt, question hidden |
| Opus 4.8 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, within-prompt, question shown |
| Opus 4.8 | proposal-preference-accuracy | 44%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.8, within-prompt, question hidden |
| Opus 4.5 | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, distinct-prompt, question shown |
| Opus 4.5 | proposal-preference-accuracy | 50%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, distinct-prompt, question hidden |
| Opus 4.5 | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, within-prompt, question shown |
| Opus 4.5 | proposal-preference-accuracy | 47%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.5, within-prompt, question hidden |
| Sonnet 5 | proposal-preference-accuracy | 54%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, distinct-prompt, question shown |
| Sonnet 5 | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, distinct-prompt, question hidden |
| Sonnet 5 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, within-prompt, question shown |
| Sonnet 5 | proposal-preference-accuracy | 42%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Sonnet 5, within-prompt, question hidden |
| Qwen3.7-Max | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, distinct-prompt, question shown |
| Qwen3.7-Max | proposal-preference-accuracy | 55%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, distinct-prompt, question hidden |
| Qwen3.7-Max | proposal-preference-accuracy | 36%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, within-prompt, question shown |
| Qwen3.7-Max | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Qwen3.7-Max, within-prompt, question hidden |
| Opus 5 | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, distinct-prompt, question shown |
| Opus 5 | proposal-preference-accuracy | 59%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, distinct-prompt, question hidden |
| Opus 5 | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, within-prompt, question shown |
| Opus 5 | proposal-preference-accuracy | 39%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 5, within-prompt, question hidden |
| Muse Spark 1.1 | proposal-preference-accuracy | 52%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, distinct-prompt, question shown |
| Muse Spark 1.1 | proposal-preference-accuracy | 54%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, distinct-prompt, question hidden |
| Muse Spark 1.1 | proposal-preference-accuracy | 35%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, within-prompt, question shown |
| Muse Spark 1.1 | proposal-preference-accuracy | 56%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Muse Spark 1.1, within-prompt, question hidden |
| Opus 4.7 | proposal-preference-accuracy | 55%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, distinct-prompt, question shown |
| Opus 4.7 | proposal-preference-accuracy | 55%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, distinct-prompt, question hidden |
| Opus 4.7 | proposal-preference-accuracy | 25%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, within-prompt, question shown |
| Opus 4.7 | proposal-preference-accuracy | 36%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.7, within-prompt, question hidden |
| GPT-5.6-luna | proposal-preference-accuracy | 54%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, distinct-prompt, question shown |
| GPT-5.6-luna | proposal-preference-accuracy | 47%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, distinct-prompt, question hidden |
| GPT-5.6-luna | proposal-preference-accuracy | 28%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, within-prompt, question shown |
| GPT-5.6-luna | proposal-preference-accuracy | 53%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-luna, within-prompt, question hidden |
| GPT-5.6-terra | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, distinct-prompt, question shown |
| GPT-5.6-terra | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, distinct-prompt, question hidden |
| GPT-5.6-terra | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, within-prompt, question shown |
| GPT-5.6-terra | proposal-preference-accuracy | 44%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-terra, within-prompt, question hidden |
| GPT-5.6-sol | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, distinct-prompt, question shown |
| GPT-5.6-sol | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, distinct-prompt, question hidden |
| GPT-5.6-sol | proposal-preference-accuracy | 28%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, within-prompt, question shown |
| GPT-5.6-sol | proposal-preference-accuracy | 50%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GPT-5.6-sol, within-prompt, question hidden |
| GLM 5.2 | proposal-preference-accuracy | 49%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, distinct-prompt, question shown |
| GLM 5.2 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, distinct-prompt, question hidden |
| GLM 5.2 | proposal-preference-accuracy | 33%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, within-prompt, question shown |
| GLM 5.2 | proposal-preference-accuracy | 39%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, GLM 5.2, within-prompt, question hidden |
| Opus 4.6 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, distinct-prompt, question shown |
| Opus 4.6 | proposal-preference-accuracy | 51%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, distinct-prompt, question hidden |
| Opus 4.6 | proposal-preference-accuracy | 31%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, within-prompt, question shown |
| Opus 4.6 | proposal-preference-accuracy | 31%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Opus 4.6, within-prompt, question hidden |
| Gemini 3.1 Pro | proposal-preference-accuracy | 47%%n = 74 preference-pairs | distinct-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, distinct-prompt, question shown |
| Gemini 3.1 Pro | proposal-preference-accuracy | 43%%n = 74 preference-pairs | distinct-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, distinct-prompt, question hidden |
| Gemini 3.1 Pro | proposal-preference-accuracy | 22%%n = 18 preference-pairs | within-prompt pairs; motivating question shown; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, within-prompt, question shown |
| Gemini 3.1 Pro | proposal-preference-accuracy | 56%%n = 18 preference-pairs | within-prompt pairs; motivating question hidden; situational framing; rubric in context; highest completed effort. Reported rounded percent, numerator not reconstructed.PDF p.38, Table 5, Gemini 3.1 Pro, within-prompt, question hidden |
| Fable 5 | proposal-preference-accuracy | 64%%n = 63 preference-pairs | dyad agrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, dyad agrees |
| Fable 5 | proposal-preference-accuracy | 56%%n = 16 preference-pairs | dyad disagrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, dyad disagrees |
| Fable 5 | proposal-preference-accuracy | 50%%n = 6 preference-pairs | dyad split; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, dyad split |
| Fable 5 | proposal-preference-accuracy | 43%%n = 7 preference-pairs | no held-out vote; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Fable 5, no held-out vote |
| Kimi K3 | proposal-preference-accuracy | 57%%n = 63 preference-pairs | dyad agrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, dyad agrees |
| Kimi K3 | proposal-preference-accuracy | 56%%n = 16 preference-pairs | dyad disagrees; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, dyad disagrees |
| Kimi K3 | proposal-preference-accuracy | 83%%n = 6 preference-pairs | dyad split; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, dyad split |
| Kimi K3 | proposal-preference-accuracy | 57%%n = 7 preference-pairs | no held-out vote; highest effort; probabilities averaged over both orderings and prompt variations.PDF p.37, Table 4, Kimi K3, no held-out vote |
| R0 | human-preference-agreement | 58%%n = 12 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R0, all solo |
| R1 | human-preference-agreement | 59%%n = 38 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R1, all solo |
| R1 | human-preference-agreement | 62%%n = 30 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R1, strong post-discussion |
| R1 | human-agreement-change | +2percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R1, delta |
| R2 | human-preference-agreement | 46%%n = 36 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R2, all solo |
| R2 | human-preference-agreement | 62%%n = 8 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R2, strong post-discussion |
| R2 | human-agreement-change | +17percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R2, delta |
| R3 | human-preference-agreement | 60%%n = 53 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R3, all solo |
| R3 | human-preference-agreement | 100%%n = 7 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R3, strong post-discussion |
| R3 | human-agreement-change | +40percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R3, delta |
| R4 | human-preference-agreement | 53%%n = 37 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R4, all solo |
| R4 | human-preference-agreement | 67%%n = 3 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R4, strong post-discussion |
| R4 | human-agreement-change | +14percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R4, delta |
| R5 | human-preference-agreement | 46%%n = 26 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R5, all solo |
| R5 | human-preference-agreement | 75%%n = 4 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R5, strong post-discussion |
| R5 | human-agreement-change | +29percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R5, delta |
| R6 | human-preference-agreement | 47%%n = 36 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R6, all solo |
| R7 | human-preference-agreement | 45%%n = 37 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R7, all solo |
| R7 | human-preference-agreement | 50%%n = 2 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R7, strong post-discussion |
| R7 | human-agreement-change | +5percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R7, delta |
| R8 | human-preference-agreement | 61%%n = 23 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R8, all solo |
| R8 | human-preference-agreement | 75%%n = 4 preference-pairs | Strong-confidence post-discussion preferences; pair count differs from baseline.PDF p.48, Table 6, R8, strong post-discussion |
| R8 | human-agreement-change | +14percentage-points | Source-computed change before rounding; not reconstructed by subtracting rounded rates.PDF p.48, Table 6, R8, delta |
| R9 | human-preference-agreement | 60%%n = 30 preference-pairs | All solo preferences; source reports rounded percent.PDF p.48, Table 6, R9, all solo |
Reading scope & review
Selected original webpage sections; linked paper, appendix and data not fully audited.
Model-assisted reading; independent evidence review pending.
Check the original source ↗