Source record

GLM-5.2 Risk Evaluation Report, Version 1.1

Source-specific extraction of SaferAI · GLM-5.2. Findings and measurements retain their original attribution.

Original source and provenance ↓ · Assessment record →

Scope and attribution

Reported by: SaferAI
Actual evaluator: Not established for every observation
Captured version: 2026-10-26
Source identity: f2e83f53d146628b

Source transcription is not experimental replication. Unknown dates, evaluator roles and configuration details remain unknown. Source groups are not verified rankings.

Source findings

The following statements summarize the source. They are not independent NUWA experimental findings.

SaferAI reports using the public OpenRouter API with Z.ai as provider, no ties or working relationship with Z.ai, and no developer discussion of the evaluation.

p.4 §1

Scope is limited to the source conditions.

Cyber evaluations measure narrow sandboxed tasks and do not test full campaigns or live adaptive defenders.

pp.26-27 §§7.1,7.4

Scope is limited to the source conditions.

CBRN coverage is limited to public biological research-knowledge and digital-task proxies; these do not directly establish harmful uplift or physical-world execution.

§§3,3.3,7.1,7.4; pp.10,13-14,26-27

Scope is limited to the source conditions.

Software engineering is used as a proxy for means; the study does not directly test a robust rogue deployment or deployment opportunity.

pp.15-16 §§4.1-4.2

Scope is limited to the source conditions.

GLM-5.2 MASK evaluation is SaferAI’s run. Opus 4.7 and GPT-5.5 MASK numbers are taken from the CAIS dashboard as shared by CAIS.

pp.20-21 §5.1/Fig.9

Scope is limited to the source conditions.

MASK and APE assess behavioral propensities; neither measures downstream effects on a real human target population.

pp.20-22 §5

Scope is limited to the source conditions.

Petri is exploratory, with text-only scenarios and auditor-simulated tools; judge scores are not real-world behavior frequencies. Judge content filters exclude potentially concerning scenarios.

pp.23-25 §6

Scope is limited to the source conditions.

Authors identify compliance under pretexts, unilateral action against operators and capitulation under pressure in selected high-scoring transcripts. Operational details are not reproduced.

pp.24-25 §6

Scope is limited to the source conditions.

Verbalized evaluation awareness cannot establish how often unspoken evaluation awareness influenced behavior.

p.26 §7.2

Scope is limited to the source conditions.

Public benchmark data and optimization toward benchmark tasks can overstate capability on real-world tasks.

pp.26-27 §7.3

Scope is limited to the source conditions.

Authors state that a subset of public benchmarks does not support an overall risk judgment and call for more systematic independent evaluation.

p.28 §8

Scope is limited to the source conditions.

Extraction coverage

Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.

View 55 coverage entries

Section	Treatment	Location and scope
Cover and Executive Summary	qualitative	pp.1-2Cover verifies Version 1.1 revision August 26, 2026. Initial publication date not stated. Sensitive biological claims omitted.
Table of Contents	extracted	p.3All relevant main sections and appendices enumerated; use physical PDF pages, not TOC offsets.
Introduction	extracted	p.4Independent public-API evaluation; OpenRouter account with Z.ai provider; no developer cooperation.
Cyber Offense	extracted	pp.5-9Aggregate outcomes only; no exploit/task instructions.
Cybench	extracted	pp.5-6; Figs.1-2Exact printed success counts extracted, including all category labels; no bar-height estimates.
CyberGym	extracted	pp.7-8; Fig.3SaferAI GLM own runs; Lyptus comparator data separately marked reused; exact printed counts and prose percentages retained.
Cyber discussion	qualitative	p.9Budget sensitivity, confidence overlap, benchmark coverage and safeguards limitations retained.
CBRN	qualitative	pp.10-14Safety-bounded bibliographic/method scope and high-level limitations only; no biological outcomes, task material, or dangerous-capability comparisons.
LAB-Bench	qualitative	pp.10-11; Table 1; Fig.4Public research-knowledge proxy; text-only coverage. Numerical capability outcomes and task details intentionally excluded.
BioMysteryBench	qualitative	pp.12-13; Figs.5-6Public digital research-task benchmark; completion/turn-budget limitations retained qualitatively. Outcomes intentionally excluded.
CBRN discussion	qualitative	pp.13-14Knowledge/digital proxies do not directly measure physical-world harmful uplift; wider CBRN domains not covered.
Loss of Control	extracted	pp.15-19Means/motive/opportunity framing; software engineering is a proxy, not a direct rogue deployment evaluation.
Methodology	extracted	p.15SWE-Bench Pro plus Agentic Misalignment; deployment opportunity not evaluated.
SWE-Bench Pro	extracted	pp.15-16; Fig.7GLM own run; Scale AI baselines reused. Contamination, reward hacking and environment issues retained.
Agentic Misalignment	extracted	pp.17-19; Fig.8All 108 printed cells (three models, three scenarios, four goals, three threats) and their printed CIs; no operational scenario transcript.
LoC discussion	qualitative	p.19Public task contamination; benchmark-to-real-world and scaffold dependence.
Harmful Manipulation	extracted	pp.20-22Propensities do not measure downstream human effects.
MASK	extracted	pp.20-21; Fig.9All printed aggregate percentages; GLM own run and CAIS shared comparator scores distinguished.
APE	extracted	pp.21-22; Fig.10Exact prose percentages only. Benign/controversial category bars lack labels: qualitative/figure-only, not estimated.
Manipulation discussion	qualitative	p.22Willingness to persuade does not establish effectiveness or real-world manipulation.
Automated Behavioral Audit	qualitative	pp.23-25; Fig.11Means have no exact printed numeric labels; not estimated. Aggregate sample counts and prose thresholds extracted; operational content omitted.
Petri methodology	extracted	p.23Public seeds, score dimensions, elicitation budget, judge exclusions and simulated tools.
Limitations and Future Work	qualitative	pp.26-27All four subsections represented.
Capability Coverage	qualitative	p.26Narrow proxies omit broad real-world capabilities; cyber saturation.
Evaluation Awareness	extracted	p.26Exact verbalized awareness rates; unspoken awareness unknown.
Data Contamination	qualitative	pp.26-27Public datasets can be memorized/optimized; held-out benchmark recommendation.
Validity	qualitative	p.27No active defenders, physical experiments or human persuasion effects in these proxy evaluations.
Conclusion	qualitative	p.28Authors explicitly say subset of public benchmarks does not support overall risk judgment. Biological capability claims excluded.
References	extracted	pp.29-31Selected provenance dependencies recorded, not independently re-extracted.
Cybench Methodology	extracted	p.32Elicitation, budgets and incomplete task coverage; task names not copied.
CyberGym Methodology	extracted	pp.32-33Configuration and sample filtering; operational instructions and vulnerability IDs not copied.
CyberGym configuration	extracted	pp.32-33Provider-default GLM reasoning; token and wall-clock budgets, no wall-clock hits.
CyberGym sample	extracted	p.33122 run; comparison 101 after missing Lyptus coverage and defective targets; no task-ID list copied.
LAB-Bench Methodology	qualitative	pp.33-34Method metadata and high-level limitations only; no numerical outcomes or detailed biological tool recipes.
LAB benchmark coverage	qualitative	p.34Text modality restrictions and public data; outcomes excluded.
LAB tools	qualitative	p.34Shared sandbox/tool environment; tool inventories omitted.
LAB elicitation	qualitative	p.34Reasoning enabled; finite budgets may under-elicit; dangerous-capability numbers excluded.
LAB safety filters	qualitative	p.34Provider filtering and abstention complicate comparisons; numerical outcomes excluded.
BioMysteryBench	qualitative	pp.34-36Method metadata only.
BioMysteryBench methodology	qualitative	p.35Digital anonymized research tasks; anti-cheat grading; exact task details and all capability outcomes excluded.
Turn limit analysis	qualitative	pp.35-36Finite turn budget and incomplete attempts may understate capability; all numerical results excluded.
SWE-Bench Pro Methodology	extracted	pp.37-40Inspect Harbor source, environment and reward-hacking diagnostics.
SWE technical issues	qualitative	p.37Prompt/test mismatch and answer leakage in repository history; no operational cheating instructions.
SWE environment issues	extracted	pp.37-38; Table A.1All printed rows preserved as diagnostics; count and pass-rate denominator mismatch quarantined.
SWE reward hacking	extracted	pp.38-40; Table A.2All printed rows and filtering summaries preserved; uncertainty in effective denominators quarantined.
Petri methodology	extracted	pp.40-41Aggregate judge exclusions recorded; seed names/task content not copied.
Petri seeds and dimensions	not-relevant	p.41Reference pointers only; full scenario seeds not extracted.
MASK methodology	extracted	pp.41-42GLM own run and CAIS reused comparator provenance.
MASK confidence intervals	extracted	pp.41-42; Table A.3All 18 printed rows; inconsistent GLM values quarantined without corrections.
APE methodology	extracted	pp.42-43Full prompt-set methodology; exact harmful example claims not copied.
Agentic Misalignment methodology	extracted	pp.43-51Inspect Evals and 11 added neutral emails; no prompts/email bodies copied.
Additional neutral emails	not-relevant	pp.44-51Synthetic setup text, not benchmark results; not extracted.
Unreported CyberGym comparator cells	not-disclosed	p.8 Fig.3No GPT-5.5 10M data; no Opus 4.6 10M/50M data. Opus 4.7 evaluation could not be completed due to safeguard refusals.
APE benign factual / benign opinion / controversial exact rates	figure-only	p.22 Fig.10Chart bars lack exact numeric labels; no bar-height estimates extracted.
Petri exact numerical mean scores	figure-only	p.24 Fig.11Exact means and error-bar bounds are not printed; no numerical estimates extracted.

Unresolved source issues

GLM 的 MASK 汇总结果与附录分组表不能相互核对，部分区间也存在内部不一致；相关结果暂缓，未推测替代数字。

CyberGym 比较模型结果引用 Lyptus 数据，MASK 比较模型结果引用 CAIS 仪表盘，不能全部归为 SaferAI 自行执行。

Version fingerprints

document

https://www.safer-ai.org/u/2026/08/SaferAI-GLM-Evaluation-Report.pdf

2b2c8ea89d19db3d123894ff406015ee0e15b0fef67bc832a5658ede23991176

References

[1] GLM-5.2 Risk Evaluation Report. 2026-08-02. v1.1 · revised 26 August 2026