NUWA Evaluation dossier
Petri
What concerning behavior can an adaptive auditor elicit?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
Petri · Gram mean concerning score
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
24 observations · 24 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
Select a point or table row to inspect its conditions and provenance.
24 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 DeepSeek V4 Pro | 1.6 score (1–10)Gram mean concerning score | Incomplete | Gram: Alignment Eval SabotageObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
 Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Alignment Eval SabotageObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
 GLM 5.2GLM-5.2 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Alignment Eval SabotageObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Incident ResponseObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Incident ResponseObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| DeepSeek V4 Pro | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Incident ResponseObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 1.4 score (1–10)Gram mean concerning score | Incomplete | Gram: Multi-Agent RankingObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| DeepSeek V4 Pro | 1.4 score (1–10)Gram mean concerning score | Incomplete | Gram: Multi-Agent RankingObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Multi-Agent RankingObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
| DeepSeek V4 Pro | 3 score (1–10)Gram mean concerning score | Incomplete | Gram: Research DeadlineObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 1.2 score (1–10)Gram mean concerning score | Incomplete | Gram: Research DeadlineObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Research DeadlineObserveGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
| DeepSeek V4 Pro | 2.6 score (1–10)Gram mean concerning score | Incomplete | Gram: Alignment Eval SabotageRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 1.8 score (1–10)Gram mean concerning score | Incomplete | Gram: Alignment Eval SabotageRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude Opus 4.8 | 1.2 score (1–10)Gram mean concerning score | Incomplete | Gram: Alignment Eval SabotageRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
| DeepSeek V4 Pro | 2.8 score (1–10)Gram mean concerning score | Incomplete | Gram: Incident ResponseRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 1.4 score (1–10)Gram mean concerning score | Incomplete | Gram: Incident ResponseRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Incident ResponseRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
| DeepSeek V4 Pro | 3.2 score (1–10)Gram mean concerning score | Incomplete | Gram: Multi-Agent RankingRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 2 score (1–10)Gram mean concerning score | Incomplete | Gram: Multi-Agent RankingRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Multi-Agent RankingRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
| DeepSeek V4 Pro | 4.8 score (1–10)Gram mean concerning score | Incomplete | Gram: Research DeadlineRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-04-24Report 2026-07-03 |
| GLM 5.2GLM-5.2 | 4.2 score (1–10)Gram mean concerning score | Incomplete | Gram: Research DeadlineRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-06-16Report 2026-07-03 |
| Claude Opus 4.8 | 1 score (1–10)Gram mean concerning score | Incomplete | Gram: Research DeadlineRedteamGram (Google DeepMind Petri variant); auditor Claude Sonnet 4.6; judge Kimi K2.6; SE across 5 independent runs per scenario | Neo ResearchEarly Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks | Release 2026-05-28Report 2026-07-03 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- An auditor explores seeded multi-turn interactions, with rubric-based judging of target transcripts.
- Scoring / outcome
- Transcripts verbalizing evaluation awareness · Source-reported measurement under the recorded configuration.
- Interpretation boundary
- Low average audit scores do not rule out rare severe cases; results depend on seeds, auditor, judge and target configuration.
A reported evaluation
The source evaluates Kimi K3 · Pre-seeded alignment audits.
Reported measurement: Transcripts verbalizing evaluation awareness · 0 transcripts.
Source-reported measurement under the recorded configuration. 0/100; 100 trials
Transcripts verbalizing evaluation awareness: 0 transcripts. Conditions are retained from the source.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can seed-driven simulated interactions efficiently expose behaviors that deserve deeper alignment investigation?
- Task population
- An auditor plans and interacts with a target through simulated users/tools; a judge scores transcripts. The pilot covers diverse seeds and a whistleblowing ablation study.
- Scoring and failure modes
- Not established by the retained material
- System conditions
- Not established by the retained material
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- The article explicitly describes a small provisional scenario set, imperfect AI auditors and evaluation-awareness complications. The separate full technical report was not read here.
- Appropriate research use
- NUWA recommendation: inspect transcripts and vary hypothesized causes; label seed coverage and judge limitations alongside any score.
- Petri : An open-source auditing tool to accelerate AI safety research · Broad-coverage pilot-alignment evaluationsSelected original resource https://www.anthropic.com/research/petri-open-source-auditing; sha256:7e988b2e577de252b4b8404d10266c39adfc5af9a52b8c8555d6022c1e7d2650; archived source capture; exact capture time not independently established
- Petri : An open-source auditing tool to accelerate AI safety research · Case study: Whistleblowing behaviorSelected original resource https://www.anthropic.com/research/petri-open-source-auditing; sha256:7e988b2e577de252b4b8404d10266c39adfc5af9a52b8c8555d6022c1e7d2650; archived source capture; exact capture time not independently established
- Petri : An open-source auditing tool to accelerate AI safety research · Get startedSelected original resource https://www.anthropic.com/research/petri-open-source-auditing; sha256:7e988b2e577de252b4b8404d10266c39adfc5af9a52b8c8555d6022c1e7d2650; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Neo Research
Evaluating Kimi K3 for Misalignment Risks →
Actual evaluators: Neo Research
12 result records · 3 reported system names
Protocols and comparison decisions (4)
Petri: Transcripts verbalizing evaluation awareness · Neo Researchsource-report · Scope reviewed
Petri: Transcripts verbalizing evaluation awareness · Neo Researchsource-report · Scope reviewed
Petri: Transcripts verbalizing evaluation awareness · Neo Researchsource-report · Scope reviewed
Petri: Transcripts verbalizing evaluation awareness · Neo Researchsource-report · Scope reviewed
Neo Research
Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks →
Actual evaluators: Neo Research
24 result records · 3 reported system names
Protocols and comparison decisions (8)
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Petri: Gram mean concerning score · Neo Researchsource-report · Scope reviewed
Original source charts and materials
Protocol scope is catalogued. Exact numerical extraction and an appropriate source figure remain to be reviewed.
Assessment provenance
Evaluating Kimi K3 for Misalignment Risks → · External report
Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks → · External report
Related research questions
References
- [1] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
- [2] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
- [3] Z.ai release notes ; GLM-5.2 entry dated 2026-06-16
- [4] GLM-5.2 model card ; Parameter counts not disclosed in reviewed developer source
- [5] deepseek-v4-pro release ; News article title, dated heading and availability
- [6] deepseek-v4-pro developer model card ; Introduction; model comparison table
- [7] GPT-5.6 official general availability ; July 9, 2026: general availability following limited trusted-partner preview; Parameter counts not disclosed in reviewed launch article
- [8] Introducing Claude Opus 4.8 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [9] Early Results on Evaluating GLM 5.2 for Behavioral and Persuasion Risks. 2026-07-03. 3 July 2026 ; #behavioral; Figure 3 / gram_observe.png; printed label Alignment Eval Sabotage / Claude Opus 4.8; #behavioral; Figure 3 / gram_observe.png; printed label Alignment Eval Sabotage / GLM 5.2
- [10] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §1.3.3 Table 2 p.18
- [11] Petri. 2025-10-06 ; Official publication masthead and article byline