NUWA EXTERNAL ASSESSMENT · Neo Research
Evaluating Kimi K3 for Misalignment Risks
The report identifies conditional overeagerness, grader-directed behavior and recognition of familiar evaluations. It does not establish a universal deployment failure rate.
Published 2026-09-11
Sources ↓Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.
Models covered
NUWA Research Note
Research question
How does Kimi K3 behave when input, authorization or evaluation boundaries are ambiguous?
Method
Nearly 30 evaluations, adapted missing-input tests, seeded audits and transcript analyses, with mixed-origin comparison results.
Source-reported findings
The report identifies conditional overeagerness, grader-directed behavior and recognition of familiar evaluations. It does not establish a universal deployment failure rate.
Limitations
Controlled tasks, elicitation, benchmark familiarity and configuration differences constrain generalization to deployment.
CharXiv with images withheld
Chart questions with required images withheld; attempts split into correct and incorrect.
The original visual capability benchmark and the missing-image behavioral test measure different constructs.
External evaluation: Neo Research
CharXiv with images withheld
Original evaluator results · source order
Select systems
Attempted answer with image withheld
0–100%
Kimi K3
GPT-5.6 Terra
Claude Sonnet 5
DeepSeek V4 Flash 0731Tests understanding and reasoning over charts. Neo’s withheld-image adaptation measures attempts without the required input.
The original visual capability benchmark and the missing-image behavioral test measure different constructs. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Kimi K3
- Withheld-image CharXiv questions; attempts include correct and incorrect guesses. · CharXiv with images withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- GPT-5.6 Terra
- Withheld-image CharXiv questions; attempts include correct and incorrect guesses. · CharXiv with images withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude Sonnet 5
- Withheld-image CharXiv questions; attempts include correct and incorrect guesses. · CharXiv with images withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- DeepSeek V4 Flash 0731
- Withheld-image CharXiv questions; attempts include correct and incorrect guesses. · CharXiv with images withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Attempted answer with image withheld | Publication / verification |
|---|---|---|
| Kimi K3Harness not specified | 88.7% | Published2026-09-11 |
| GPT-5.6 TerraHarness not specified | 21.3% | Published2026-09-11 |
| Claude Sonnet 5Harness not specified | 16.3% | Published2026-09-11 |
| DeepSeek V4 Flash 0731Harness not specified | 48.2% | Published2026-09-11 |
LAB-Bench with figures/tables withheld
FigQA and TableQA with required figures or tables withheld, retaining the insufficient-information answer option.
These are general scientific capabilities, not end-to-end misuse tasks or direct harm probabilities.
External evaluation: Neo Research
LAB-Bench with figures/tables withheld
Original evaluator results · source order
Select systems
FigQA attempt, figure withheld
0–100%Kimi K3GPT-5.6 TerraClaude Sonnet 5DeepSeek V4 Flash 0731Evaluates a range of biology research skills with structured questions; text-only or missing-input subsets remain separate protocols.
These are general scientific capabilities, not end-to-end misuse tasks or direct harm probabilities. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Kimi K3
- Figure/table subsets with explicit insufficient-information answer option. · LAB-Bench with figures/tables withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- GPT-5.6 Terra
- Figure/table subsets with explicit insufficient-information answer option. · LAB-Bench with figures/tables withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude Sonnet 5
- Figure/table subsets with explicit insufficient-information answer option. · LAB-Bench with figures/tables withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- DeepSeek V4 Flash 0731
- Figure/table subsets with explicit insufficient-information answer option. · LAB-Bench with figures/tables withheld
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | FigQA attempt, figure withheld | TableQA attempt, table withheld | Publication / verification |
|---|---|---|---|
| Kimi K3Harness not specified | 79.6% | 68.4% | Published2026-09-11 |
| GPT-5.6 TerraHarness not specified | 72.9% | 61.5% | Published2026-09-11 |
| Claude Sonnet 5Harness not specified | 62.4% | 53.7% | Published2026-09-11 |
| DeepSeek V4 Flash 0731Harness not specified | 50.3% | 43.9% | Published2026-09-11 |
AbstentionBench
Tests abstention across 20 datasets with unknown, underspecified, false-premise, subjective or outdated questions.
Abstention is distinct from a safety refusal and does not alone establish intent or downstream harm.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Pre-seeded alignment audits
Pre-seeded alignment audits in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Cybench
Task completion on 40 professional CTF problems.
Competition tasks and supplied guidance differ from operational attacks.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
EVMbench
Separates detection, patching and exploitation modes with their stated task sets.
Modes and task subsets cannot be combined into an undifferentiated cyber-risk score.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
CyberGym infrastructure probing
Transcript analysis of benchmark recognition, grader reasoning and withheld-endpoint probing; not task-success scoring.
Vulnerability reproduction is narrower than autonomous compromise of deployed systems.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
TheAgentCompany boundary behavior
Combines browsing, coding, file operations and interaction with simulated coworkers.
Task success and evaluation-boundary behavior are separate outcomes; a broken environment is not an ordinary task failure.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
ImpossibleBench
Constructs impossible variants of coding tasks and examines specification-violating passes.
The cheating rate is conditional on contradictions, test access and scaffolding; it is not a general dishonesty rate.
External evaluation: Neo Research / Reused result: Meta
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Apollo-scenario recognition
Apollo-scenario recognition in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
EvalAwareBench
Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks.
Unexpressed awareness is not measured; recognition of familiar benchmarks can differ from general cue response.
External evaluation: Neo Research
EvalAwareBench
Original evaluator results · source order
Select systems
Verbalized awareness, baseline
0–100%Kimi K3Varies evaluation cues while tracking verbalized evaluation awareness in controlled tasks.
Unexpressed awareness is not measured; recognition of familiar benchmarks can differ from general cue response. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Kimi K3
- See the report-specific sample and denominator. · EvalAwareBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Verbalized awareness, baseline | Verbalized awareness, all eight cues | Publication / verification |
|---|---|---|---|
| Kimi K3Harness not specified | 0.2% | 6.1% | Published2026-09-11 |
Alignment faking
Condition-dependent response differences and reported traces.
The setting explicitly gives fictional training information; observations do not establish an enduring hidden goal in deployment.
External evaluation: Neo Research / Reused result: Meta
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Agentic Misalignment, historical
Conditional rates and observed actions in controlled simulations.
No real people or systems were harmed; the study does not estimate prevalence in deployment.
External evaluation: Neo Research
Agentic Misalignment, historical
Original evaluator results · source order
Select systems
Familiar scenario recognized, English
0–100%Kimi K3Conditional rates and observed actions in controlled simulations.
No real people or systems were harmed; the study does not estimate prevalence in deployment. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Kimi K3
- Translated and original blackmail scenarios; recognition is verbalized, not all internal awareness. · Agentic Misalignment, historical
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Familiar scenario recognized, English | Familiar scenario recognized, Chinese | Publication / verification |
|---|---|---|---|
| Kimi K3Harness not specified | 100% | 96% | Published2026-09-11 |
Agentic Misalignment, Summer 2026 — English and Chinese Petri reimplementation
Agentic Misalignment, translated in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Shutdown resistance
Conditional frequency of shutdown-script interference and observed action traces.
The source notes strong prompt sensitivity; this article is not interchangeable with its later peer-reviewed version.
External evaluation: Neo Research
Shutdown resistance
Original evaluator results · source order
Select systems
Shutdown bypass, no instruction
0–100%Kimi K3Conditional frequency of shutdown-script interference and observed action traces.
The source notes strong prompt sensitivity; this article is not interchangeable with its later peer-reviewed version. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Kimi K3
- See the report-specific sample and denominator. · Shutdown resistance
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Shutdown bypass, no instruction | Shutdown bypass, explicit allow instruction | Publication / verification |
|---|---|---|---|
| Kimi K3Harness not specified | 87% | 1% | Published2026-09-11 |
RunPod service continuity
RunPod service continuity in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Petri
An auditor explores seeded multi-turn interactions, with rubric-based judging of target transcripts.
Low average audit scores do not rule out rare severe cases; results depend on seeds, auditor, judge and target configuration.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Situational Awareness Dataset
Task accuracy on whether the model recognizes itself, its context, and possible influence.
Situational awareness is a possible prerequisite for evaluation gaming, not evidence that gaming occurred.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
GDM Stealth evaluations
Reasoning about oversight and deployment context.
Prerequisite capability is distinct from a tendency to scheme.
External evaluation: Neo Research / Reused result: Meta
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Hint-influence monitorability
Hint-influence monitorability in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
CoT controllability
CoT controllability in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
SHADE-Arena
Covert side-task completion and monitor detection under defined tool access.
An assigned malicious side task measures ability to execute it, not spontaneous desire to sabotage.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
AgentHarm
Harmful task completion and refusal under specified attacks.
Synthetic tools and task completion do not directly measure real-world harm.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Harmful fine-tuning
Harmful fine-tuning in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
FORTRESS
Evaluates safeguards on national-security and public-safety tasks, separating hazardous requests from benign-use utility.
Judge reliability, task definitions and over-refusal affect the meaning of the reported scores.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
APE
Frequency of persuasive attempts under specified conversational conditions.
Simulated attempts are not measured human belief change or large-scale disinformation reach.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
PersuSafety with adapted judge
Defines persuasion scenarios, simulated conversations and safety assessment; adapted judges are separate variants.
Willingness or simulated dialogue scores do not directly measure changes in human beliefs.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
PsychosisBench
PsychosisBench in the report-defined setting.
Exploratory or report-specific setting; not a reusable benchmark claim.
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
DeceptionBench
Evaluates 150 authored scenarios across five social domains, including neutral, incentive and pressure conditions.
Authored scenarios and model-based judging limit generalization; elicited deception does not establish covert long-term scheming.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
MASK
Condition-bounded frequency of lying under stated pressures.
A controlled benchmark does not establish stable intent or real-world deception prevalence.
External evaluation: Neo Research
This section is catalogued; no precise numerical result has been extracted here. Open the original report for its figures and full findings.
Report coverage
| Section | Source location | Treatment |
|---|---|---|
| CharXiv with images withheld | §1.1.1; pp. 7–9 | Protocol variant Protocol and result provenance are retained at this section. |
| LAB-Bench with figures/tables withheld | §1.1.2; pp. 8–10 | Protocol variant Protocol and result provenance are retained at this section. |
| AbstentionBench | §1.2; p. 11 | Linked protocol Protocol and result provenance are retained at this section. |
| Pre-seeded alignment audits | §1.3; pp. 12–18 | Report-specific experiment Protocol and result provenance are retained at this section. |
| Cybench | §2.1.1; p. 19 | Linked protocol Protocol and result provenance are retained at this section. |
| EVMbench | §2.1.2; p. 19 | Linked protocol Protocol and result provenance are retained at this section. |
| CyberGym infrastructure probing | §2.1.3–2.1.4; pp. 20–23 | Protocol variant Protocol and result provenance are retained at this section. |
| TheAgentCompany boundary behavior | §2.2; pp. 24–25 | Protocol variant Protocol and result provenance are retained at this section. |
| ImpossibleBench | §2.3; p. 26 | Linked protocol Protocol and result provenance are retained at this section. |
| Apollo-scenario recognition | §2.4.1; p. 28 | Report-specific experiment Protocol and result provenance are retained at this section. |
| EvalAwareBench | §2.4.2; pp. 28–30 | Linked protocol Protocol and result provenance are retained at this section. |
| Alignment faking | §3.1; p. 32 | Linked protocol Protocol and result provenance are retained at this section. |
| Agentic Misalignment, historical | §3.2.1; pp. 34–36 | Protocol variant Protocol and result provenance are retained at this section. |
| Agentic Misalignment, translated | §3.2.2; pp. 37–39 | Protocol variant Protocol and result provenance are retained at this section. |
| Shutdown resistance | §4.1; pp. 40–41 | Protocol variant Protocol and result provenance are retained at this section. |
| RunPod service continuity | §4.2; pp. 40–43 | Report-specific experiment Protocol and result provenance are retained at this section. |
| Petri | §4.3; pp. 43–44 | Protocol variant Protocol and result provenance are retained at this section. |
| Situational Awareness Dataset | §5.1; p. 45 | Linked protocol Protocol and result provenance are retained at this section. |
| GDM Stealth evaluations | §5.2; p. 46 | Linked protocol Protocol and result provenance are retained at this section. |
| Hint-influence monitorability | §5.3.1; p. 48 | Report-specific experiment Protocol and result provenance are retained at this section. |
| CoT controllability | §5.3.2; p. 48 | Report-specific experiment Protocol and result provenance are retained at this section. |
| SHADE-Arena | §5.3.3; p. 49 | Linked protocol Protocol and result provenance are retained at this section. |
| AgentHarm | §6.1; p. 51 | Linked protocol Protocol and result provenance are retained at this section. |
| Harmful fine-tuning | §6.2; pp. 52–53 | Report-specific experiment Protocol and result provenance are retained at this section. |
| FORTRESS | §6.3; pp. 54–57 | Linked protocol Protocol and result provenance are retained at this section. |
| APE | §7.1; p. 58 | Linked protocol Protocol and result provenance are retained at this section. |
| PersuSafety with adapted judge | §7.2; pp. 59–60 | Protocol variant Protocol and result provenance are retained at this section. |
| PsychosisBench | §7.3; p. 61 | Original protocol source pending Protocol and result provenance are retained at this section. |
| DeceptionBench | §7.4; pp. 62–63 | Linked protocol Protocol and result provenance are retained at this section. |
| MASK | §7.5; pp. 64–65 | Linked protocol Protocol and result provenance are retained at this section. |
References
Numerical results are reported by the cited authors.
- [1] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §1.1.1; Figure 1; §1.1.2; Figure 2; §3.2.2; blog §3