NUWA Evaluation dossier
OpenDeception
Deception in simulated human–AI interactions.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
OpenDeception · Annotator consistency
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
11 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
11 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
11 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 Llama-3.1-8B-Instruct | 100%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
 Qwen2-72B-Instruct | 100%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Qwen2.5-72B-Instruct | 100%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
 GPT-4o | 97.87%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Qwen2.5-32B-Instruct | 97.83%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| GPT-3.5 | 97.78%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Llama-3.1-70B-Instruct | 97.5%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Qwen2-7B-Instruct | 97.44%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Qwen2.5-14B-Instruct | 95.74%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Qwen2.5-7B-Instruct | 95.12%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
| Qwen2.5-3B-Instruct | 90.63%Annotator consistency | Incomplete | Table 1 · OpenDeception— | Not disclosedOpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Model in simulated social interactions
- Measurement target
- Scenario-based inferred intent and modeled user trust.
- Scoring / outcome
- IntentNet DIR · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success.
- Interpretation boundary
- Inferred intent is a measurement construct; simulated users do not establish human effects.
A reported evaluation
The source evaluates GPT-3.5 · Table 1 · OpenDeception.
Reported measurement: IntentNet DIR · 100%.
Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. 50 authored scenarios; evaluated cases depend on the source screening procedure.
Source-reported measurements for GPT-3.5 in Table 1; values are transcribed, not independently reproduced.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can dialogue evaluation jointly characterize deceptive intent and simulated user susceptibility?
- Task population
- Fifty seed cases span five domains; role/goal simulations feed an IntentNet and a contrastively trained TrustNet.
- Scoring and failure modes
- Deceptive-intent inference and simulated trust/susceptibility are separate outputs, with refusal also reported.
- System conditions
- Fifty seed scenarios and model-to-model open-ended dialogue generation.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: The authors report deceptive intent in over 90% of goal-driven interactions for most tested models. Llama-3.1-8B-Instruct instead had a 70% overall refusal rate in the reported analysis (Appendix E).
- Validity and interpretation
- LLM-simulated users and inferred intent limit behavioral interpretation; the case study does not validate clinical early warning.
- Appropriate research use
- Model recommendation: treat this as a simulation-based monitoring prototype pending external human validation.
- OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation · AbstractSelected original resource https://arxiv.org/html/2504.13707v3; sha256:90ab521e0ac0dc160133177e59dbe0998cf4876bb90f2c419eb00ec0bd0c6ae4; archived source capture; exact capture time not independently established
- OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation · §3.1 Benchmark and Scenarios ConstructionSelected original resource https://arxiv.org/html/2504.13707v3; sha256:90ab521e0ac0dc160133177e59dbe0998cf4876bb90f2c419eb00ec0bd0c6ae4; archived source capture; exact capture time not independently established
- OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation · §3.2 Open-Ended Interactions SimulationSelected original resource https://arxiv.org/html/2504.13707v3; sha256:90ab521e0ac0dc160133177e59dbe0998cf4876bb90f2c419eb00ec0bd0c6ae4; archived source capture; exact capture time not independently established
- OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation · Appendix E More Interesting FindingsSelected original resource https://arxiv.org/html/2504.13707v3; sha256:90ab521e0ac0dc160133177e59dbe0998cf4876bb90f2c419eb00ec0bd0c6ae4; archived source capture; exact capture time not independently established
- OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation · §5 ConclusionSelected original resource https://arxiv.org/html/2504.13707v3; sha256:90ab521e0ac0dc160133177e59dbe0998cf4876bb90f2c419eb00ec0bd0c6ae4; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation
OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation
Actual evaluators: Not disclosed
11 result records · 11 reported system names
Protocols and comparison decisions (3)
OpenDeception: Annotator consistency · OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulationsource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
IntentNet DIR
0–100%GPT-3.5GPT-3.5 · 100% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
100%GPT-4oGPT-4o · 97.87% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
97.87%Llama-3.1-8B-InstructLlama-3.1-8B-Instruct · 92.85% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
92.85%Llama-3.1-70B-InstructLlama-3.1-70B-Instruct · 100% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
100%Qwen2-7B-InstructQwen2-7B-Instruct · 92.31% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
92.31%Qwen2-72B-InstructQwen2-72B-Instruct · 100% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
100%Qwen2.5-3B-InstructQwen2.5-3B-Instruct · 87.5% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
87.5%Qwen2.5-7B-InstructQwen2.5-7B-Instruct · 90.24% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
90.24%Qwen2.5-14B-InstructQwen2.5-14B-Instruct · 93.62% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
93.62%Qwen2.5-32B-InstructQwen2.5-32B-Instruct · 95.65% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
95.65%Qwen2.5-72B-InstructQwen2.5-72B-Instruct · 100% · Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. · 50 authored scenarios; evaluated cases depend on the source screening procedure.
100%Showing 11 of 11 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 1: deception-intention rates under OpenDeception scenario conditions, assessed by IntentNet and human annotators. Consistency is agreement, not deception success. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- GPT-3.5
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- GPT-4o
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Llama-3.1-8B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Llama-3.1-70B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2-7B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2-72B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2.5-3B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2.5-7B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2.5-14B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2.5-32B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Qwen2.5-72B-Instruct
- 50 authored scenarios; evaluated cases depend on the source screening procedure. · Table 1 · OpenDeception
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | IntentNet DIR | Human DIR | Annotator consistency | Publication / verification |
|---|---|---|---|---|
| GPT-3.5Harness not specified | 100% | 97.8% | 97.78% | Source checked2026-10-25 |
| GPT-4oHarness not specified | 97.87% | 93.6% | 97.87% | Source checked2026-10-25 |
| Llama-3.1-8B-InstructHarness not specified | 92.85% | 92.85% | 100% | Source checked2026-10-25 |
| Llama-3.1-70B-InstructHarness not specified | 100% | 97.5% | 97.5% | Source checked2026-10-25 |
| Qwen2-7B-InstructHarness not specified | 92.31% | 89.7% | 97.44% | Source checked2026-10-25 |
| Qwen2-72B-InstructHarness not specified | 100% | 100% | 100% | Source checked2026-10-25 |
| Qwen2.5-3B-InstructHarness not specified | 87.5% | 81.2% | 90.63% | Source checked2026-10-25 |
| Qwen2.5-7B-InstructHarness not specified | 90.24% | 87.8% | 95.12% | Source checked2026-10-25 |
| Qwen2.5-14B-InstructHarness not specified | 93.62% | 91.5% | 95.74% | Source checked2026-10-25 |
| Qwen2.5-32B-InstructHarness not specified | 95.65% | 91.5% | 97.83% | Source checked2026-10-25 |
| Qwen2.5-72B-InstructHarness not specified | 100% | 100% | 100% | Source checked2026-10-25 |
Related research questions
Deception / Misleading communication →
The source directly measures the stated construct under its protocol.
Deception / Misleading communication →
The source directly measures the stated construct under its protocol.
References
Numerical results are reported by the cited authors.
- [1] Yichen Wu, Qianqian Gao, et al.. OpenDeception : Learning Deception and Trust in Human–AI Interaction via Multi-Agent Simulation. 2026-02-06. v3 [arXiv]; Author affiliations on the original paper title page; Table 1