Evaluation dossier
PersuSafety
How does model engagement with persuasion requests vary with topic harm?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
PersuSafety · Turn-1 persuasion attempt rate
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
12 observations · 9 dated
3 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
12 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
| 100%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationLow harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 | |
| 89%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationLow harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-06-30Report 2026-09-11 | |
| 86%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationLow harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-09Report 2026-09-11 | |
| 75%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationLow harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 | |
| 84%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationMedium harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 | |
| 30%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationMedium harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 | |
| 19%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationMedium harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-06-30Report 2026-09-11 | |
| 5%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationMedium harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-09Report 2026-09-11 | |
| 86%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationHigh harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release —Report 2026-09-11 | |
| 31%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationHigh harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-16Report 2026-09-11 | |
| 17%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationHigh harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-06-30Report 2026-09-11 | |
| 0%Turn-1 persuasion attempt rate | Incomplete | PersuSafety open-ended single-turn adaptationHigh harm— | Neo ResearchEvaluating Kimi K3 for Misalignment Risks | Release 2026-07-09Report 2026-09-11 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Defines persuasion scenarios, simulated conversations and safety assessment; adapted judges are separate variants.
- Scoring / outcome
- Medium-harm attempt rate · Defines persuasion scenarios, simulated conversations and safety assessment; adapted judges are separate variants.
- Interpretation boundary
- Willingness or simulated dialogue scores do not directly measure changes in human beliefs.
Defines persuasion scenarios, simulated conversations and safety assessment; adapted judges are separate variants.
A reported evaluation
The source evaluates GLM-5.2 · PersuSafety adapted free-generation judge.
Reported measurement: Medium-harm attempt rate · 54%.
Defines persuasion scenarios, simulated conversations and safety assessment; adapted judges are separate variants. Free-form responses scored by an APE-style StrongREJECT judge, not the original binary willingness setup.
Source-reported measurements in §2; Figure 6. Transcription is not independent experimental replication.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Do models reject unethical persuasion and avoid unethical strategies after accepting a task?
- Task population
- A three-stage framework generates tasks, simulates conversations and scores six harmful topics/fifteen strategies; eight persuaders use GPT-4o as default simulated persuadee.
- Scoring and failure modes
- Refusal and unethical-strategy scores; strategy analysis conditions on nonrefused tasks.
- System conditions
- Eight persuader models; default simulated GPT-4o persuadee and vulnerability profiles.
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- Simulated persuadees do not establish human impact; refusal-conditioned strategy scores cannot be treated as unconditional population rates.
- Appropriate research use
- Model recommendation: report refusal and conditional strategy use side by side, preserving profile visibility.
- PersuSafety · §3.3 Benchmarked LLMs and SetupSelected original resource https://arxiv.org/html/2504.10430; sha256:b97a60a11e25b2c2f1283620ee8ca36bb2c583e7fba5f06d6982564475a973f4; archived source capture; exact capture time not independently established
- PersuSafety · §4 Experiments and Discussions: denominator noteSelected original resource https://arxiv.org/html/2504.10430; sha256:b97a60a11e25b2c2f1283620ee8ca36bb2c583e7fba5f06d6982564475a973f4; archived source capture; exact capture time not independently established
- PersuSafety · §4.2 How LLM Persuaders Exploit Persuadee’s Vulnerabilities?Selected original resource https://arxiv.org/html/2504.10430; sha256:b97a60a11e25b2c2f1283620ee8ca36bb2c583e7fba5f06d6982564475a973f4; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Neo Research
Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks →
Actual evaluators: Neo Research
9 result records · 4 reported system names
Protocols and comparison decisions (6)
Neo Research
Evaluating Kimi K3 for Misalignment Risks →
Actual evaluators: Neo Research
12 result records · 4 reported system names
Protocols and comparison decisions (3)
Neo Research
Evaluating DeepSeek v4 Pro for Frontier Risks →
Actual evaluators: Neo Research · Not disclosed
17 result records · 6 reported system names
Protocols and comparison decisions (12)
Original source charts and materials
PersuSafety adapted free-generation judge
Original evaluator results · source order
Select systems
Medium-harm attempt rate
0–100%Defines persuasion scenarios, simulated conversations and safety assessment; adapted judges are separate variants.
Willingness or simulated dialogue scores do not directly measure changes in human beliefs. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- GLM-5.2
- Free-form responses scored by an APE-style StrongREJECT judge, not the original binary willingness setup. · PersuSafety adapted free-generation judge
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- Claude Sonnet 5
- Free-form responses scored by an APE-style StrongREJECT judge, not the original binary willingness setup. · PersuSafety adapted free-generation judge
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- DeepSeek V4 Pro
- Free-form responses scored by an APE-style StrongREJECT judge, not the original binary willingness setup. · PersuSafety adapted free-generation judge
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Medium-harm attempt rate | High-harm attempt rate | Publication / verification |
|---|---|---|---|
| 54% | 6% | Published2026-07-03 | |
| 57% | 31% | Published2026-07-03 | |
| 76% | 78% | Published2026-07-03 |
Assessment provenance
Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks → · External report
Evaluating Kimi K3 for Misalignment Risks → · External report
Evaluating DeepSeek v4 Pro for Frontier Risks → · External report
Related research questions
Disinformation / Persuasion attempts →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
- [2] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
- [3] Z.ai release notes ; GLM-5.2 entry dated 2026-06-16
- [4] GLM-5.2 model card ; Parameter counts not disclosed in reviewed developer source
- [5] deepseek-v4-pro release ; News article title, dated heading and availability
- [6] deepseek-v4-pro developer model card ; Introduction; model comparison table
- [7] GPT-5.6 official general availability ; July 9, 2026: general availability following limited trusted-partner preview; Parameter counts not disclosed in reviewed launch article
- [8] Introducing Claude Sonnet 5 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [9] Gemini 2.5: Our newest Gemini model with thinking ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [10] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §7.2 Figure 40 p.60
- [11] Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks. 2026-07-03. 3 July 2026 ; §2; Figure 6; #persuasion-benchmarks; Figure 6 caption and following two paragraphs
- [12] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 14; PersuSafety single turn; refusal rate; PDF p. 15; PersuSafety single turn; acceptance rate; PDF p. 15; PersuSafety single turn; accepted low-harm count
- [13] PersuSafety. 2025-04-14