NUWA analysis · model-assisted · independent review pending

Compute budget changes the cyber capability observed

Monitor model capability together with task scope and inference resources. A higher observed result can reflect a different budget or harness as well as a different model.

Evidence snapshot 2026-10-28.2 · 2026-10-27

Evidence and argument

SaferAI’s report describes CyberGym as vulnerability reproduction and selects hard Level 1 tasks following Lyptus. It starts from a 122-task sample and retains 101 tasks with available comparison data. Its source prose reports GLM-5.2 success rising from 36.6% at 2M tokens to 76.2% at 50M; the extracted condition records retain their own printed precision. SaferAI executed the GLM runs, while the GPT-5.5 and Opus 4.6 comparison values were calculated from Lyptus data. Shared task identity does not erase execution differences. The source’s overlapping Wilson intervals do not prove equivalence.

The same model changes substantially with budget

SaferAI reports GLM-5.2 CyberGym reproduction rates of 36.6% at 2M tokens and 76.2% at 50M tokens on its hard-task subset.

Source finding

2 supporting observations

GLM-5.2 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label
GLM-5.2 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label

GLM-5.2 Risk Evaluation Report · §2.2–2.3, PDF pp.7–9; Figure 3; §2.3 PDF p.9 explicitly reports 36.6% and 76.2%; Figure 3 counts retained separately

A common task subset helps explain comparisons

The 101-task comparison separates SaferAI’s GLM-5.2 runs from Lyptus’s GPT-5.5 and Opus 4.6 runs. Their execution settings remain attached.

Source finding

6 supporting observations

GLM-5.2 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label
GLM-5.2 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label
GLM-5.2 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label
GPT-5.5 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label
GPT-5.5 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label
Claude Opus 4.6 · CyberGym Level 1 hard-task matched subset · p.8 Fig.3 printed bar label

GLM-5.2 Risk Evaluation Report · SaferAI §2.2, pp.7–8; common101tasks; each plotted run retains execution origin

Lyptus Cyber Task Horizons: aggregate model runs · SaferAI §2.2, pp.7–8; common101tasks; each plotted run retains execution origin

Reproduction and attack progression answer different questions

CyberGym evidence complements Cybench and AgentCyberRange. Differences in task scope and attempt rules belong in separate views.

NUWA interpretation

CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · CyberGym §3.2; Cybench §2.4; AgentCyberRange适用范围

Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · CyberGym §3.2; Cybench §2.4; AgentCyberRange适用范围

AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks · CyberGym §3.2; Cybench §2.4; AgentCyberRange适用范围

Limits and remaining assumptions

Source reports overlapping 95% Wilson intervals; exact endpoints are not transcribed because the figure does not print them. This is a task-matched descriptive comparison, not a strict matched model ranking.

Next research question

Resolve protocol fields across reports first. Use matched comparisons only after verifying model identity, task version, attempts, scaffold and budget.

References

[1] GLM-5.2 Risk Evaluation Report. 2026-08-02. v1.1 · revised 26 August 2026 ; §2.2–2.3, PDF pp.7–9; Figure3
[2] Lyptus Cyber Task Horizons: aggregate model runs ; Source data used by SaferAI for comparison values