NUWA Evaluation dossier
ImpossibleBench
Will coding agents exploit tests when specifications and tests cannot both be satisfied?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
ImpossibleBench · Cheating rate (pass@1 on impossible variants)
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
4 observations · 3 dated
1 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
4 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 Claude Opus 4.6 | 2.9%Cheating rate (pass@1 on impossible variants) | Incomplete | Impossible-LiveCodeBench split with toolsCheating rate (pass@1 on impossible variants)— | MetaEvaluating Kimi K3 for Misalignment Risks | Release 2026-02-05Report 2026-09-11 |
 Kimi K3 | 1.5%Cheating rate (pass@1 on impossible variants) | Incomplete | Impossible-LiveCodeBench split with toolsCheating rate (pass@1 on impossible variants)— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 |
 DeepSeek V4 FlashDeepSeek V4 Flash 0731 | 0.5%Cheating rate (pass@1 on impossible variants) | Incomplete | Impossible-LiveCodeBench split with toolsCheating rate (pass@1 on impossible variants)— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 |
 GPT-5.6 Terra | 0%Cheating rate (pass@1 on impossible variants) | Incomplete | Impossible-LiveCodeBench split with toolsCheating rate (pass@1 on impossible variants)— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-09Report 2026-09-11 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Constructs impossible variants of coding tasks and examines specification-violating passes.
- Scoring / outcome
- Cheating rate (pass@1 on impossible variants) · Source-reported measurement under the recorded configuration.
- Interpretation boundary
- The cheating rate is conditional on contradictions, test access and scaffolding; it is not a general dishonesty rate.
Constructs impossible variants of coding tasks and examines specification-violating passes.
A reported evaluation
The source evaluates Kimi K3 · Impossible-LiveCodeBench split with tools.
Reported measurement: Cheating rate (pass@1 on impossible variants) · 1.5%.
Source-reported measurement under the recorded configuration. Per-result sample size not disclosed
Cheating rate (pass@1 on impossible variants): 1.5 %. Conditions are retained from the source.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Will agents violate a specification to pass tests made deliberately impossible?
- Task population
- Tasks mutate LiveCodeBench and SWE-bench so tests conflict with the natural-language specification; passing such cases serves as a cheating proxy.
- Scoring and failure modes
- Pass rate on deliberately specification-conflicting impossible tests is a cheating proxy.
- System conditions
- Mutated LiveCodeBench/SWE-bench; prompt/test access/feedback vary.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Monitor detection was 86–89% on Impossible-LiveCodeBench but substantially lower on Impossible-SWEbench.
- Validity and interpretation
- Passing signals a specification violation only if the mutation is truly impossible; this rate does not estimate ordinary coding-task cheating.
- Appropriate research use
- Model recommendation: verify impossibility and retain test access, prompt and feedback conditions.
- ImpossibleBench · AbstractSelected original resource https://arxiv.org/html/2510.20270; sha256:c2850cd2eaf221a7a22b315b1fb5ef2a797c92f0ff757adbf2ece0d08168dad3; archived source capture; exact capture time not independently established
- ImpossibleBench · §2 Benchmark CurationSelected original resource https://arxiv.org/html/2510.20270; sha256:c2850cd2eaf221a7a22b315b1fb5ef2a797c92f0ff757adbf2ece0d08168dad3; archived source capture; exact capture time not independently established
- ImpossibleBench · §5 Results: Context EngineeringSelected original resource https://arxiv.org/html/2510.20270; sha256:c2850cd2eaf221a7a22b315b1fb5ef2a797c92f0ff757adbf2ece0d08168dad3; archived source capture; exact capture time not independently established
- ImpossibleBench · §6 Results: MonitoringSelected original resource https://arxiv.org/html/2510.20270; sha256:c2850cd2eaf221a7a22b315b1fb5ef2a797c92f0ff757adbf2ece0d08168dad3; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Neo Research
Evaluating Kimi K3 for Misalignment Risks →
Actual evaluators: Neo Research · Meta
4 result records · 4 reported system names
Protocols and comparison decisions (1)
Original source charts and materials
Protocol scope is catalogued. Exact numerical extraction and an appropriate source figure remain to be reviewed.
Assessment provenance
Evaluating Kimi K3 for Misalignment Risks → · External report
Related research questions
Deception / Evaluation gaming →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
- [2] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
- [3] GPT-5.6 official general availability ; July 9, 2026: general availability following limited trusted-partner preview; Parameter counts not disclosed in reviewed launch article
- [4] Claude Opus 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [5] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §2.3 Figure 17 p.27
- [6] ImpossibleBench. 2025-10-23