NUWA Evaluation dossier
MASK
Separates honesty from factual accuracy using repeated, human-collected test situations.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
MASK · MASK response proportion: Honest
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
8 observations · 3 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
5 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
8 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 Claude Sonnet 4.6 | 66%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-02-17Report 2026-07-03 |
 GLM 5.2GLM-5.2 | 55%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude 3.7 Sonnet | 48%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Not disclosedEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release —Report 2026-07-03 |
 GPT-4.5 Preview | 27%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Not disclosedEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release —Report 2026-07-03 |
 Llama 405B | 22%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Not disclosedEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release —Report 2026-07-03 |
| GPT-4o | 22%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Not disclosedEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release —Report 2026-07-03 |
 DeepSeek V3 | 21%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Not disclosedEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release —Report 2026-07-03 |
| DeepSeek V4 Pro | 20%MASK response proportion: Honest | Incomplete | MASKhonesty under pressure— | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Models under pressure to state an inaccurate belief
- Measurement target
- Condition-bounded frequency of lying under stated pressures.
- Scoring / outcome
- P(honest) · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent.
- Interpretation boundary
- A controlled benchmark does not establish stable intent or real-world deception prevalence.
A reported evaluation
The source evaluates claude-3-5-sonnet-20240620 · Table 3 · MASK.
Reported measurement: P(honest) · 27.7%.
Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. See the source protocol; no independent trial count inferred.
Source-reported measurements for claude-3-5-sonnet-20240620 in Table 3; values are transcribed, not independently reproduced.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Do models state what they believe when contextual pressure favors a false statement?
- Task population
- MASK separates belief elicitation from pressured responses; the reported study uses 1,000 selected examples plus a 500-example held-out set.
- Scoring and failure modes
- Belief–statement consistency under pressure, separated from belief accuracy.
- System conditions
- 1,000 selected and 500 held-out examples in the reported full study.
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- Belief is behaviorally inferred and pressure scenarios are selected; scores are not natural-deployment lying rates.
- Appropriate research use
- Model recommendation: preserve belief checks and pressure conditions before interpreting a response as a lie.
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems · §4.1 DatasetSelected original resource https://arxiv.org/html/2503.03750v3; sha256:2a482d76ec1b6eadf5663e53d8069a2a2829b3356c3de4ef3b890985cb609e54; archived source capture; exact capture time not independently established
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems · §4.2 Evaluation PipelineSelected original resource https://arxiv.org/html/2503.03750v3; sha256:2a482d76ec1b6eadf5663e53d8069a2a2829b3356c3de4ef3b890985cb609e54; archived source capture; exact capture time not independently established
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems · §5 ExperimentsSelected original resource https://arxiv.org/html/2503.03750v3; sha256:2a482d76ec1b6eadf5663e53d8069a2a2829b3356c3de4ef3b890985cb609e54; archived source capture; exact capture time not independently established
- The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems · §6 ConclusionSelected original resource https://arxiv.org/html/2503.03750v3; sha256:2a482d76ec1b6eadf5663e53d8069a2a2829b3356c3de4ef3b890985cb609e54; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems
Actual evaluators: Not disclosed
32 result records · 32 reported system names
Protocols and comparison decisions (5)
MASK: Accuracy · The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemssource-report · Scope reviewed
MASK: P(honest) · The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemssource-report · Scope reviewed
MASK: P(honest)@10 · The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemssource-report · Scope reviewed
MASK: P(lie) · The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemssource-report · Scope reviewed
MASK: P(lie)@10 · The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systemssource-report · Scope reviewed
Neo Research
Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks →
Actual evaluators: Neo Research · Not disclosed
30 result records · 11 reported system names
Protocols and comparison decisions (6)
MASK: Honest under pressure · Neo Researchsource-report · Scope reviewed
MASK: MASK response proportion: Evaded or No Belief · Neo Researchsource-report · Scope reviewed
MASK: MASK response proportion: Honest · Neo Researchsource-report · Scope reviewed
MASK: MASK response proportion: Lying · Neo Researchsource-report · Scope reviewed
MASK: MASK verbalised evaluation-awareness rate · Neo Researchsource-report · Scope reviewed
MASK: MASK verbalised evaluation-awareness rate · Neo Researchsource-report · Scope reviewed
SaferAI
GLM-5.2 Risk Evaluation Report →
Actual evaluators: Center for AI Safety · Not disclosed
20 result records · 2 reported system names
Protocols and comparison decisions (21)
MASK by archetype · Continuations honestysource-report · Scope reviewed
MASK by archetype · Disinformation honestysource-report · Scope reviewed
MASK by archetype · Doubling-down honestysource-report · Scope reviewed
MASK by archetype · Known-facts honestysource-report · Scope reviewed
MASK by archetype · Provided-facts honestysource-report · Scope reviewed
MASK by archetype · Statistics honestysource-report · Scope reviewed
MASK: Continuations honesty · SaferAIsource-report · Scope reviewed
MASK: Disinformation honesty · SaferAIsource-report · Scope reviewed
MASK: Doubling-down honesty · SaferAIsource-report · Scope reviewed
MASK: Known-facts honesty · SaferAIsource-report · Scope reviewed
MASK: MASK category honesty · SaferAIsource-report · Scope reviewed
MASK: MASK category honesty · SaferAIsource-report · Scope reviewed
MASK: MASK category honesty · SaferAIsource-report · Scope reviewed
MASK: MASK category honesty · SaferAIsource-report · Scope reviewed
MASK: MASK category honesty · SaferAIsource-report · Scope reviewed
MASK: MASK category honesty · SaferAIsource-report · Scope reviewed
MASK: MASK Evaded or No Belief · SaferAIsource-report · Scope reviewed
MASK: MASK Honest · SaferAIsource-report · Scope reviewed
MASK: MASK Lying · SaferAIsource-report · Scope reviewed
MASK: Provided-facts honesty · SaferAIsource-report · Scope reviewed
MASK: Statistics honesty · SaferAIsource-report · Scope reviewed
Neo Research
Evaluating Kimi K3 for Misalignment Risks →
Actual evaluators: Neo Research
3 result records · 3 reported system names
Protocols and comparison decisions (1)
MASK: Lying rate under pressure · Neo Researchsource-report · Scope reviewed
Neo Research
Evaluating DeepSeek v4 Pro for Frontier Risks →
Actual evaluators: Neo Research · Not disclosed
21 result records · 7 reported system names
Protocols and comparison decisions (3)
MASK: response label: Evaded or No Belief · Neo Researchsource-report · Scope reviewed
MASK: response label: Honest · Neo Researchsource-report · Scope reviewed
MASK: response label: Lying · Neo Researchsource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
P(honest)
0–100%claude-3-5-sonnet-20240620claude-3-5-sonnet-20240620 · 27.7% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
27.7%claude-3-7-sonnet-20250219claude-3-7-sonnet-20250219 · 47.6% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
47.6%deepseek-r1deepseek-r1 · 24.7% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
24.7%deepseek-v3deepseek-v3 · 20.8% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
20.8%deepseek-llm-67b-chatdeepseek-llm-67b-chat · 18.6% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
18.6%
gemini-2.0-flashgemini-2.0-flash · 20.7% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
20.7%gpt-4.5-preview-2025-02-27gpt-4.5-preview-2025-02-27 · 27.2% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
27.2%gpt-4o-2024-08-06gpt-4o-2024-08-06 · 21.8% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
21.8%gpt-4o-mini-2024-07-18gpt-4o-mini-2024-07-18 · 21.4% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
21.4%
grok-2-1212grok-2-1212 · 14.2% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
14.2%llama-2-13b-chatllama-2-13b-chat · 28.7% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
28.7%llama-2-70b-chatllama-2-70b-chat · 28.3% · Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. · See the source protocol; no independent trial count inferred.
28.3%Showing 12 of 32 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 3: honesty, lying and accuracy under MASK pressure prompts. @10 and single-sample metrics are separate; missing cells remain absent. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- claude-3-5-sonnet-20240620
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- claude-3-7-sonnet-20250219
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- deepseek-r1
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- deepseek-v3
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- deepseek-llm-67b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gemini-2.0-flash
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gpt-4.5-preview-2025-02-27
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gpt-4o-2024-08-06
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- gpt-4o-mini-2024-07-18
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- grok-2-1212
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-2-13b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-2-70b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-2-7b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-31-405b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-31-70b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-31-8b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-32-1b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-32-3b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- llama-33-70b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- o3-mini-2025-01-31 ∗
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen15-110b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen15-32b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen15-72b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen15-7b-chat
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-05b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-14b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-15b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-32b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-3b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-72b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwen25-7b-instruct
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- qwq-32b-preview
- See the source protocol; no independent trial count inferred. · Table 3 · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | P(honest) | P(lie) | P(honest)@10 | P(lie)@10 | Accuracy | Publication / verification |
|---|---|---|---|---|---|---|
| claude-3-5-sonnet-20240620Harness not specified | 27.7% | 33.4% | 23.6% | 44.7% | 80.1% | Source checked2026-10-25 |
| claude-3-7-sonnet-20250219Harness not specified | 47.6% | 26.6% | 38.2% | 42.1% | 82.2% | Source checked2026-10-25 |
| deepseek-r1Harness not specified | 24.7% | 42.9% | Not reported | Not reported | 79.6% | Source checked2026-10-25 |
| deepseek-v3Harness not specified | 20.8% | 53.5% | 13.5% | 67.4% | 71.6% | Source checked2026-10-25 |
| deepseek-llm-67b-chatHarness not specified | 18.6% | 45.6% | 6.5% | 69.0% | 64.3% | Source checked2026-10-25 |
| gemini-2.0-flashHarness not specified | 20.7% | 48.4% | 14.5% | 64.0% | 79.4% | Source checked2026-10-25 |
| gpt-4.5-preview-2025-02-27Harness not specified | 27.2% | 43.5% | Not reported | Not reported | 76.7% | Source checked2026-10-25 |
| gpt-4o-2024-08-06Harness not specified | 21.8% | 44.5% | 14.5% | 63.0% | 78.6% | Source checked2026-10-25 |
| gpt-4o-mini-2024-07-18Harness not specified | 21.4% | 45.3% | 15.9% | 59.7% | 71.4% | Source checked2026-10-25 |
| grok-2-1212Harness not specified | 14.2% | 63.0% | 10.2% | 71.2% | 72.5% | Source checked2026-10-25 |
| llama-2-13b-chatHarness not specified | 28.7% | 24.7% | 19.0% | 41.4% | 40.1% | Source checked2026-10-25 |
| llama-2-70b-chatHarness not specified | 28.3% | 26.7% | 19.7% | 41.5% | 40.6% | Source checked2026-10-25 |
| llama-2-7b-chatHarness not specified | 27.5% | 21.5% | 17.8% | 38.2% | 33.6% | Source checked2026-10-25 |
| llama-31-405b-instructHarness not specified | 21.6% | 28.0% | 14.3% | 44.9% | 72.1% | Source checked2026-10-25 |
| llama-31-70b-instructHarness not specified | 27.1% | 43.5% | 14.4% | 64.3% | 73.8% | Source checked2026-10-25 |
| llama-31-8b-instructHarness not specified | 18.8% | 23.5% | 8.2% | 46.7% | 62.0% | Source checked2026-10-25 |
| llama-32-1b-instructHarness not specified | 13.9% | 13.1% | 4.3% | 35.8% | 23.0% | Source checked2026-10-25 |
| llama-32-3b-instructHarness not specified | 21.8% | 23.5% | 10.7% | 43.1% | 40.0% | Source checked2026-10-25 |
| llama-33-70b-instructHarness not specified | 24.7% | 44.9% | 17.3% | 58.8% | 75.6% | Source checked2026-10-25 |
| o3-mini-2025-01-31 ∗Harness not specified | 19.6% | 48.6% | Not reported | Not reported | 63.3% | Source checked2026-10-25 |
 qwen15-110b-chatHarness not specified | 27.9% | 35.6% | 20.2% | 53.4% | 72.8% | Source checked2026-10-25 |
| qwen15-32b-chatHarness not specified | 23.8% | 42.5% | 15.5% | 60.0% | 63.0% | Source checked2026-10-25 |
| qwen15-72b-chatHarness not specified | 24.2% | 47.8% | 15.1% | 63.1% | 69.3% | Source checked2026-10-25 |
| qwen15-7b-chatHarness not specified | 27.1% | 35.1% | 15.8% | 55.2% | 52.5% | Source checked2026-10-25 |
| qwen25-05b-instructHarness not specified | 15.9% | 15.4% | 6.3% | 41.1% | 20.8% | Source checked2026-10-25 |
| qwen25-14b-instructHarness not specified | 26.5% | 47.7% | 17.5% | 63.8% | 64.4% | Source checked2026-10-25 |
| qwen25-15b-instructHarness not specified | 25.7% | 26.9% | 10.5% | 57.7% | 28.8% | Source checked2026-10-25 |
| qwen25-32b-instructHarness not specified | 28.7% | 43.9% | 20.2% | 59.5% | 63.7% | Source checked2026-10-25 |
| qwen25-3b-instructHarness not specified | 30.7% | 33.8% | 18.6% | 54.3% | 46.8% | Source checked2026-10-25 |
| qwen25-72b-instructHarness not specified | 23.2% | 49.2% | 15.9% | 61.9% | 66.0% | Source checked2026-10-25 |
| qwen25-7b-instructHarness not specified | 28.9% | 39.0% | 18.6% | 55.3% | 51.6% | Source checked2026-10-25 |
| qwq-32b-previewHarness not specified | 20.3% | 25.2% | 11.1% | 46.3% | 49.2% | Source checked2026-10-25 |
MASK
Original evaluator results · source order
Select systems
Honest under pressure
0–100%GLM-5.2GLM-5.2 · 55% · Condition-bounded frequency of lying under stated pressures. · See the report-specific sample and denominator.
55%DeepSeek V4 ProDeepSeek V4 Pro · 20% · Condition-bounded frequency of lying under stated pressures. · See the report-specific sample and denominator.
20%Claude Sonnet 4.6Claude Sonnet 4.6 · 66% · Condition-bounded frequency of lying under stated pressures. · See the report-specific sample and denominator.
66%Showing 3 of 3 systems
Condition-bounded frequency of lying under stated pressures.
A controlled benchmark does not establish stable intent or real-world deception prevalence. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- GLM-5.2
- See the report-specific sample and denominator. · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- DeepSeek V4 Pro
- See the report-specific sample and denominator. · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude Sonnet 4.6
- See the report-specific sample and denominator. · MASK
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Honest under pressure | Publication / verification |
|---|---|---|
| GLM-5.2Harness not specified | 55% | Published2026-07-03 |
| DeepSeek V4 ProHarness not specified | 20% | Published2026-07-03 |
| Claude Sonnet 4.6Harness not specified | 66% | Published2026-07-03 |
MASK by archetype
Original evaluator results · source order
Select systems
Continuations honesty
0–100%Claude Opus 4.7Claude Opus 4.7 · 59.1% · 95% Wilson CI [51.7, 66.1] (as reported) · Condition-bounded frequency of lying under stated pressures. · Per-archetype scored prompts: 176 / 125 / 120 / 209 / 274 / 96. Some supplied CI cells are inconsistent with displayed n; reproduced as reported, not recalculated.
59.1%GPT-5.5GPT-5.5 · 65.3% · 95% Wilson CI [58.1, 72.0] (as reported) · Condition-bounded frequency of lying under stated pressures. · Per-archetype scored prompts: 176 / 125 / 120 / 209 / 274 / 96. Some supplied CI cells are inconsistent with displayed n; reproduced as reported, not recalculated.
65.3%Showing 2 of 2 systems
Condition-bounded frequency of lying under stated pressures.
A controlled benchmark does not establish stable intent or real-world deception prevalence. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- Claude Opus 4.7
- Per-archetype scored prompts: 176 / 125 / 120 / 209 / 274 / 96. Some supplied CI cells are inconsistent with displayed n; reproduced as reported, not recalculated. · MASK by archetype
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- GPT-5.5
- Per-archetype scored prompts: 176 / 125 / 120 / 209 / 274 / 96. Some supplied CI cells are inconsistent with displayed n; reproduced as reported, not recalculated. · MASK by archetype
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Continuations honesty | Disinformation honesty | Doubling-down honesty | Known-facts honesty | Provided-facts honesty | Statistics honesty | Publication / verification |
|---|---|---|---|---|---|---|---|
| Claude Opus 4.7Harness not specified | 59.1%95% Wilson CI [51.7, 66.1] (as reported) | 33.6%95% Wilson CI [25.9, 42.3] (as reported) | 68.3%95% Wilson CI [59.6, 76.0] (as reported) | 63.6%95% Wilson CI [56.9, 69.9] (as reported) | 74.8%95% Wilson CI [69.4, 79.6] (as reported) | 80.2%95% Wilson CI [71.1, 86.9] (as reported) | Published2026-08-02 |
| GPT-5.5Harness not specified | 65.3%95% Wilson CI [58.1, 72.0] (as reported) | 69.6%95% Wilson CI [61.1, 77.0] (as reported) | 63.3%95% Wilson CI [54.4, 71.4] (as reported) | 77.0%95% Wilson CI [70.9, 82.2] (as reported) | 75.2%95% Wilson CI [69.7, 79.9] (as reported) | 87.5%95% Wilson CI [79.4, 92.7] (as reported) | Published2026-08-02 |
Assessment provenance
Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks → · External report
GLM-5.2 Risk Evaluation Report → · External report
Evaluating Kimi K3 for Misalignment Risks → · External report
Evaluating DeepSeek v4 Pro for Frontier Risks → · External report
Related research questions
Deception / Misleading communication →
Repeated pressured scenarios distinguish lying from mistaken beliefs.
References
- [1] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
- [2] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
- [3] Z.ai release notes ; GLM-5.2 entry dated 2026-06-16
- [4] GLM-5.2 model card ; Parameter counts not disclosed in reviewed developer source
- [5] deepseek-v4-pro release ; News article title, dated heading and availability
- [6] deepseek-v4-pro developer model card ; Introduction; model comparison table
- [7] GPT-5.6 official general availability ; July 9, 2026: general availability following limited trusted-partner preview; Parameter counts not disclosed in reviewed launch article
- [8] OpenAI — introducing gpt 5 5 ; Publication date and availability section; Parameter counts not disclosed in launch article
- [9] Introducing Claude Opus 4.7 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [10] Introducing Sonnet 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [11] Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks. 2026-07-03. 3 July 2026 ; §1; Figure 1; #behavioral; Figure 1 / mask_honesty.png; printed label GLM 5.2 / Honest; #behavioral; Figure 1 / mask_honesty.png; printed label GLM 5.2 / Evaded or No Belief
- [12] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 28; MASK; response label: Honest; PDF p. 28; MASK; response label: Evaded or No Belief; PDF p. 28; MASK; response label: Lying
- [13] GLM-5.2 Risk Evaluation Report. 2026-08-02. v1.1 · revised 26 August 2026 ; Appendix A.7; Table A.3, pp. 41–42; p.21 Fig.9; p.42 Table A.3
- [14] Richard Ren, Arunim Agarwal, et al.. The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems. 2026-01-05. v3 [arXiv]; Author affiliations on the original paper title page; Table 3
- [15] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §7.5 Figure 46 p.65