NUWA Evaluation dossier
EvalAwareBench
Which evaluation cues cause models to verbalize awareness of being tested?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
EvalAwareBench · Recognition
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
8 observations · 4 dated
4 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
8 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 DeepSeek V4 FlashDeepSeek V4 Flash 0731 | 1.6%Recognition | Incomplete | EvalAwareBench Safety tasksBaseline— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 |
 Kimi K3 | 0.2%Recognition | Incomplete | EvalAwareBench Safety tasksBaseline— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 |
| Kimi K3 | 7.2%Recognition | Incomplete | EvalAwareBench Safety tasksAll-8— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 |
| DeepSeek V4 FlashDeepSeek V4 Flash 0731 | 6.4%Recognition | Incomplete | EvalAwareBench Safety tasksAll-8— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 |
| DeepSeek V4 FlashDeepSeek V4 Flash 0731 | 1%Recognition | Incomplete | EvalAwareBench Capability tasksBaseline— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 |
| Kimi K3 | 0.2%Recognition | Incomplete | EvalAwareBench Capability tasksBaseline— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 |
| DeepSeek V4 FlashDeepSeek V4 Flash 0731 | 7.4%Recognition | Incomplete | EvalAwareBench Capability tasksAll-8— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 |
| Kimi K3 | 5%Recognition | Incomplete | EvalAwareBench Capability tasksAll-8— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks.
- Scoring / outcome
- Verbalized awareness, baseline · Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks.
- Interpretation boundary
- Unexpressed awareness is not measured; recognition of familiar benchmarks can differ from general cue response.
Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks.
A reported evaluation
The source evaluates Kimi K3 · EvalAwareBench.
Reported measurement: Verbalized awareness, baseline · 0.2%.
Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks. See the report-specific sample and denominator.
Source-reported measurements in §2.4.2; blog §3. Transcription is not independent experimental replication.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Which evaluation cues trigger recognition, and does recognition change behavior?
- Task population
- Eight controlled factors vary across 100 paired safety/capability tasks; nine models and four earlier benchmarks provide additional ten-repeat tests.
- Scoring and failure modes
- Verbalized recognition and behavioral compliance are measured separately under paired cue interventions.
- System conditions
- 100 paired tasks, eight factors; nine-model/four-benchmark earlier tests use ten repetitions.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Some factors reduce capability-task performance without matching evaluation-awareness signals, indicating difficulty confounding (§4.2).
- Validity and interpretation
- Measurement targets verbalized recognition under black-box access; unspoken awareness and CoT unfaithfulness remain gaps (§5).
- Appropriate research use
- Model recommendation: compare paired cue interventions and preserve non-awareness pathways before attributing score changes to evaluation gaming.
- EvalAwareBench · AbstractSelected original resource https://arxiv.org/html/2605.23055; sha256:f034653da02841f088ba9201e5a4731153c049f4c09d5d90e02780492917e38b; archived source capture; exact capture time not independently established
- EvalAwareBench · §3.1 SetupSelected original resource https://arxiv.org/html/2605.23055; sha256:f034653da02841f088ba9201e5a4731153c049f4c09d5d90e02780492917e38b; archived source capture; exact capture time not independently established
- EvalAwareBench · §4.2 ResultsSelected original resource https://arxiv.org/html/2605.23055; sha256:f034653da02841f088ba9201e5a4731153c049f4c09d5d90e02780492917e38b; archived source capture; exact capture time not independently established
- EvalAwareBench · §5 Discussion, Safety Implication, and Future WorkSelected original resource https://arxiv.org/html/2605.23055; sha256:f034653da02841f088ba9201e5a4731153c049f4c09d5d90e02780492917e38b; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Neo Research
Evaluating Kimi K3 for Misalignment Risks →
Actual evaluators: Neo Research
13 result records · 2 reported system names
Protocols and comparison decisions (8)
Original source charts and materials
EvalAwareBench
Original evaluator results · source order
Select systems
Verbalized awareness, baseline
0–100%Kimi K3Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks.
Unexpressed awareness is not measured; recognition of familiar benchmarks can differ from general cue response. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Kimi K3
- See the report-specific sample and denominator. · EvalAwareBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Verbalized awareness, baseline | Verbalized awareness, all eight cues | Publication / verification |
|---|---|---|---|
| Kimi K3Harness not specified | 0.2% | 6.1% | Published2026-09-11 |
Assessment provenance
Evaluating Kimi K3 for Misalignment Risks → · External report
Related research questions
Deception / Evaluation gaming →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
- [2] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
- [3] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §2.4.2; blog §3; §2.4.2 Figure 19 p.29; §2.4.2 Table 6 p.30
- [4] EvalAwareBench