NUWA Evidence record
DeepSeek V4 Flash
ImpossibleBench · Evidence record
Source snapshot, chart and review scope →
Original source charts and materials
Recorded measurements
- Cheating rate (pass@1 on impossible variants)
- 0.5%
Per-result sample size not disclosed
Harness not specified · Budget not specified
Assessment provenance
Evaluating Kimi K3 for Misalignment Risks → · External report
Actual evaluation: Neo Research · Model record →
Method and scope
Constructs impossible variants of coding tasks and examines specification-violating passes.
- Evidence type
- Benchmark
- Assessment subject
- The model or agent in the stated task configuration.
- Primary dimension
- Propensity
- Evidence snapshot
- 2026-10-29.1
The main dimension follows the stated measurement; contextual risk relevance does not establish harmful deployment capability.
Source finding
Cheating rate (pass@1 on impossible variants): 0.5 %. Conditions are retained from the source.
Limitations
The cheating rate is conditional on contradictions, test access and scaffolding; it is not a general dishonesty rate.
Only LiveCodeBench split, not Impossible-SWEbench. Figure does not expose separate oneoff/conflicting breakdown or denominator.
Related research questions
Deception / Evaluation gaming →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] ImpossibleBench. 2025-10-23
- [2] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §2.3 Figure 17 p.27