NUWA Risk questions
Cybersecurity
How might AI systems perform or enable cyber operations, and how can agents be compromised?
Cybersecurity
Cyber task performance across models, budgets and evaluation protocols.
PUBLISHED EVIDENCE
CyberGym · Vulnerability reproduction rate
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
105 observations · 43 dated
- CyberGym Level 1 hard-task matched subset
- CyberGym hard subset with token budgets
- level1
- level0
- level2
- level3
62 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
105 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
  Multi-model (DeepSeek-V4.1-Flash, Abliterated GLM-5.2) · Lyrie Agent | 99.2%Success rate | Incomplete | level1score_10Lyrie Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
 Multi-model (Creation Model, DeepSeek-V4-Pro, Qwen 3.8 Max) · Creation \(天工\) | 98.5%Success rate | Incomplete | level1score_10Creation \(天工\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Multi-model (DeepSeek-V4-Flash, OrcaCyber-Zero-1.0) · OrcaCyber | 98%Success rate | Incomplete | level1score_10OrcaCyber | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4.1-Flash · RO0T Agent | 97.7%Success rate | Incomplete | level1score_10RO0T Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.3 · Sangfor AI | 97.2%Success rate | Incomplete | level1score_10Sangfor AI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Flash · Ostorlab NeutronDeepSeek-V4-Flash | 97.1%Success rate | Incomplete | level1score_10Ostorlab Neutron | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| DeepSeek-V4-Flash · VARAS-OneMind \(智安·宇盾\)DeepSeek-V4-Flash | 97%Success rate | Incomplete | level1score_10VARAS-OneMind \(智安·宇盾\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| GLM-5.3 · Alipay AI4SDL | 96.8%Success rate | Incomplete | level1score_10Alipay AI4SDL | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Multi-model (DeepSeek-V4-Flash, GLM-5.2, GLM-5.3, GLM-5.3-Flash) · NPC Agent System | 96.7%Success rate | Incomplete | level1score_10NPC Agent System | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Hero (finetuned from GLM-5.2) · Hero Agent | 96.6%Success rate | Incomplete | level1score_10Hero Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Pro · Panshi \(磐石\)DeepSeek V4 Pro | 95.4%Success rate | Incomplete | level1score_10Panshi \(磐石\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| Multi-model (DeepSeek-V4-Pro, Creation Model, Qwen 3.8 Max) · Creation \(天工\) | 95.3%Success rate | Incomplete | level1score_10Creation \(天工\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.3 · DoGNAVY | 95%Success rate | Incomplete | level1score_10DoGNAVY | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Multi-model (DeepSeek-V4-Flash, DeepSeek-V4.1-Flash, GLM-5.3, GLM-5.3-Flash) · ai-deep-audit | 94.2%Success rate | Incomplete | level1score_10ai-deep-audit | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Flash · RO0T AgentDeepSeek-V4-Flash | 94.2%Success rate | Incomplete | level1score_10RO0T Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| DeepSeek-V4-Flash · Sangfor AIDeepSeek-V4-Flash | 93.2%Success rate | Incomplete | level1score_10Sangfor AI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| DeepSeek-V4-Flash · PwnBotDeepSeek-V4-Flash | 93.1%Success rate | Incomplete | level1score_10PwnBot | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| DeepSeek-V4.1-Flash · AiPy Mini | 92.8%Success rate | Incomplete | level1score_10AiPy Mini | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Multi-model (DeepSeek-V4-Flash, GLM-5.2) · Ji Realm \(极境\) | 92.4%Success rate | Incomplete | level1score_10Ji Realm \(极境\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| JT236B · JiuXuan \(九玄\) | 92.3%Success rate | Incomplete | level1score_10JiuXuan \(九玄\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Qwen-Internal · Spur | 91.8%Success rate | Incomplete | level1score_10Spur | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Flash · NASHDeepSeek-V4-Flash | 91.4%Success rate | Incomplete | level1score_10NASH | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
 Multi-model (Grok 4.5, Grok 4.6) · Gcsa Agent | 91.3%Success rate | Incomplete | level1score_10Gcsa Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Flash · FangcunCyberDeepSeek-V4-Flash | 91.3%Success rate | Incomplete | level1score_10FangcunCyber | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| DeepSeek-V4-Flash · Whitzard \(白泽\)DeepSeek-V4-Flash | 91.2%Success rate | Incomplete | level1score_10Whitzard \(白泽\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
 Claude Opus 4.6 · mnemeClaude Opus 4.6 | 91%Success rate | Incomplete | level1score_10mneme | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-02-05Report — |
 Multi-model (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6) · MDASH | 91%Success rate | Incomplete | level1score_10MDASH | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Multi-model (GPT-5.5, Claude Opus 4.6) · Wiz Atlas | 90.9%Success rate | Incomplete | level1score_10Wiz Atlas | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.2 · DoGNAVYGLM-5.2 | 90.8%Success rate | Incomplete | level1score_10DoGNAVY | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-06-16Report — |
| DeepSeek-V4-Pro · HiAUDIT AgentDeepSeek V4 Pro | 90.6%Success rate | Incomplete | level1score_10HiAUDIT Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| Claude Opus 4.6 · CrystallineClaude Opus 4.6 | 89.6%Success rate | Incomplete | level1score_10Crystalline | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-02-05Report — |
| DeepSeek-V4-Flash · TriLaneDeepSeek-V4-Flash | 88.9%Success rate | Incomplete | level1score_10TriLane | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| XekRung-1.5-27B-Preview (finetuned from Qwen3.8-27B) · XekRung Agent | 88.9%Success rate | Incomplete | level1score_10XekRung Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Multi-model (GPT-5.4, Claude Opus 4.6, Claude Sonnet 4.6) · MDASH | 88.4%Success rate | Incomplete | level1score_10MDASH | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-5.5 | 88.12%CyberGym pass@1 | Incomplete | CyberGym Level 1 hard-task matched subsetCyberGym pass@150M tokens per task— | Lyptus ResearchGLM-5.2 Risk Evaluation Report | Release 2026-04-23Report 2026-08-02 |
| DeepSeek-V4-Flash · GusionDeepSeek-V4-Flash | 87.1%Success rate | Incomplete | level1score_10Gusion | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
 Gemini 3.8 Flash Cyber · Google Antigravity | 86.3%Success rate | Incomplete | level1score_10Google Antigravity | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.2 · RedbudAIGLM-5.2 | 86.3%Success rate | Incomplete | level1score_10RedbudAI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-06-16Report — |
| GLM-5.2 · Sangfor AIGLM-5.2 | 86.3%Success rate | Incomplete | level1score_10Sangfor AI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-06-16Report — |
| Multi-model (MAI-Cyber-1-Flash, GPT-5.4, Claude Opus 4.6) · MDASH | 86.3%Success rate | Incomplete | level1score_10MDASH | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| ASL-Cyber-Flash (finetuned from DeepSeek-V4-Flash) · Claude Code | 85.9%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-5.5-Cyber · OpenAI Agent | 85.6%Success rate | Incomplete | level1score_10OpenAI Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| XekRung · Velldepth Agent | 85.3%Success rate | Incomplete | level1score_10Velldepth Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.2 · Xuanwu Atuin AIGLM-5.2 | 84.8%Success rate | Incomplete | level1score_10Xuanwu Atuin AI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-06-16Report — |
| GLM-5.3 · Claude Code | 84.5%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Feyospace-v1.1 (finetuned from Qwen3.8-27B) · Claude Code | 84.4%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.1 · Xuanwu Atuin AI | 84%Success rate | Incomplete | level1score_10Xuanwu Atuin AI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Pro · DeepSeek AgentDeepSeek V4 Pro | 83.3%Success rate | Incomplete | level1score_10DeepSeek Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| Claude Mythos Preview · Anthropic AgentClaude Mythos Preview | 83.1%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-5.5 · OpenAI AgentGPT-5.5 | 81.8%Success rate | Incomplete | level1score_10OpenAI Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-23Report — |
| Basin (basin1-0912-pre-dpo) · Codex CLI | 80%Success rate | Incomplete | level1score_10Codex CLI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Grok 4.6 · Grok Build | 79.7%Success rate | Incomplete | level1score_10Grok Build | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Grok 4.5 · Grok Build | 79%Success rate | Incomplete | level1score_10Grok Build | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-5.4 · OpenAI AgentGPT-5.4 | 79%Success rate | Incomplete | level1score_10OpenAI Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-03-05Report — |
| DeepSeek-V4-Flash · DeepSeek AgentDeepSeek-V4-Flash | 76.7%Success rate | Incomplete | level1score_10DeepSeek Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-24Report — |
| GLM-5.2 | 76.24%CyberGym pass@1 | Incomplete | CyberGym Level 1 hard-task matched subsetprovider-default reasoning50M tokens per taskInspect Evals | SaferAIGLM-5.2 Risk Evaluation Report | Release 2026-06-16Report 2026-08-02 |
| GPT-5.4 · VARAS-OneMind \(智安·宇盾\)GPT-5.4 | 73.8%Success rate | Incomplete | level1score_10VARAS-OneMind \(智安·宇盾\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-03-05Report — |
 MiniMax M3 · MopMonk Agent | 73.1%Success rate | Incomplete | level1score_10MopMonk Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.1 · JiuXuan \(九玄\) | 72.9%Success rate | Incomplete | level1score_10JiuXuan \(九玄\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.1-FP8 · Whitzard \(白泽\) | 68.9%Success rate | Incomplete | level1score_10Whitzard \(白泽\) | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.1 · Claude Code | 68.7%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Sonnet 4.5 · Anthropic Agent | 66.7%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Opus 4.6 · Anthropic AgentClaude Opus 4.6 | 66.6%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-02-05Report — |
| GPT-5.4 · Codex CLIGPT-5.4 | 66.3%Success rate | Incomplete | level1score_10Codex CLI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-03-05Report — |
| Claude Sonnet 4.6 · Anthropic AgentClaude Sonnet 4.6 | 65.2%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-02-17Report — |
| DeepSeek-V4-Pro Preview · XD Agent | 64.1%Success rate | Incomplete | level1score_10XD Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GLM-5.2 | 63.37%CyberGym pass@1 | Incomplete | CyberGym Level 1 hard-task matched subsetprovider-default reasoning10M tokens per taskInspect Evals | SaferAIGLM-5.2 Risk Evaluation Report | Release 2026-06-16Report 2026-08-02 |
| Feyospace-v1 · Claude Code | 63%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Opus 4.1 · Anthropic AgentClaude Opus 4.1 | 61.3%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-08-05Report — |
| GPT-5 · SageAgentGPT-5 | 60.2%Success rate | Incomplete | level1score_10SageAgent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-08-07Report — |
| Claude Sonnet 4 · Anthropic Agent | 59.5%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
 Muse Spark 1.1 (helpful-only) · Meta Agent | 59%Success rate | Incomplete | level1score_10Meta Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V4-Pro Preview · Claude Code | 57.7%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-5.5 | 55.45%CyberGym pass@1 | Incomplete | CyberGym Level 1 hard-task matched subsetCyberGym pass@12M tokens per task— | Lyptus ResearchGLM-5.2 Risk Evaluation Report | Release 2026-04-23Report 2026-08-02 |
| Feyospace-v0 · Claude Code | 51.2%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Opus 4.5 · Anthropic AgentClaude Opus 4.5 | 50.6%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-11-24Report — |
| Claude Sonnet 3.7 · Anthropic AgentClaude 3.7 Sonnet | 47.2%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-02-24Report — |
| Claude Opus 4.6 | 44.55%CyberGym pass@1 | Incomplete | CyberGym Level 1 hard-task matched subsetCyberGym pass@12M tokens per task— | Lyptus ResearchGLM-5.2 Risk Evaluation Report | Release 2026-02-05Report 2026-08-02 |
| Muse Spark · Meta AgentMuse Spark | 43.5%Success rate | Incomplete | level1score_10Meta Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-04-08Report — |
| GLM-5 · Claude CodeGLM-5 | 43.2%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-02-12Report — |
 Kimi K2.5 · Kimi Agent | 41.3%Success rate | Incomplete | level1score_10Kimi Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-5 · OpenHandsGPT-5 | 39.4%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-08-07Report — |
| Gemini 3.1 Pro · Gemini CLIGemini 3.1 Pro | 38.8%Success rate | Incomplete | level1score_10Gemini CLI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2026-02-19Report — |
| GLM-5.2 | 36.63%CyberGym pass@1 | Incomplete | CyberGym Level 1 hard-task matched subsetprovider-default reasoning2M tokens per taskInspect Evals | SaferAIGLM-5.2 Risk Evaluation Report | Release 2026-06-16Report 2026-08-02 |
| Claude Sonnet 4.5 · Anthropic Agent | 28.9%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Opus 4.1 · Anthropic AgentClaude Opus 4.1 | 25%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-08-05Report — |
| GLM-4.7 · Claude Code | 23.5%Success rate | Incomplete | level1score_10Claude Code | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Sonnet 4 · Anthropic Agent | 22.6%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Sonnet 4 · OpenHands | 17.9%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-4.1 · OpenHands | 17.1%Success rate | Incomplete | level3score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Sonnet 3.7 · Anthropic AgentClaude 3.7 Sonnet | 14.5%Success rate | Incomplete | level1score_10Anthropic Agent | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-02-24Report — |
| GPT-4.1 · OpenHands | 13.1%Success rate | Incomplete | level2score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Claude Sonnet 3.7 · OpenHandsClaude 3.7 Sonnet | 11.9%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release 2025-02-24Report — |
| GPT-4.1 · OpenHands | 9.4%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-4.1 · Cybench | 9%Success rate | Incomplete | level1score_10Cybench | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-4.1 · Codex CLI | 7.4%Success rate | Incomplete | level1score_10Codex CLI | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-4.1 · ENiGMA | 7.2%Success rate | Incomplete | level1score_10ENiGMA | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Gemini 2.5 Flash Preview · OpenHands | 4.8%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| DeepSeek-V3 · OpenHands | 3.6%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| GPT-4.1 · OpenHands | 3.5%Success rate | Incomplete | level0score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| o4-mini · OpenHands | 2.5%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| R2E-Gym-32B · OpenHands | 2%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| Qwen3-235B-A22B · OpenHands | 1.9%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| OpenHands-LM-32B · OpenHands | 1.7%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
| SWE-Gym-32B · OpenHands | 0.1%Success rate | Incomplete | level1score_10OpenHands | Not disclosedCyberGym official leaderboard and machine-data extraction | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →New research and interpretation
Selected original sections · model-assisted reading; independent review pending. These records add context without changing trend scores.
- Incident Report: unsanctioned agent behaviour during cyber testing →
The report documents an evaluation incident under permissive conditions, not a population estimate of commercial-agent harm.
- Scaling Misuse Detection with Self-Assessment →
Teacher-relative recall, passive monitoring and adaptive blocking are distinct endpoints.
- TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents →
Execution-based traces reveal intermediate violations that endpoint scoring misses.
- Frontier AI Trends Report →
Cross-domain trends require methodology-specific interpretation and negative-evidence limits.
Scope
Offensive task capability and agent compromise are tracked separately; authorized security research is not observed criminal use.
Threat model
A specified actor uses an AI system, access, tools, and targets in a cyber task.
Evidence gaps
Benchmark success cannot establish real-world attack prevalence or expected losses.
Research on this topic
AgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges
QuestionCan an agent discover an attack surface from sparse entry information and continue making decisions as access and information accumulate?
Source summaryThe introduction explicitly distinguishes vulnerability reproduction from autonomous range progression: the starting information and task structures differ (适用范围).
NUWA research note · independent review pending · Contains: question / method / limits
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models
QuestionHow capable are language-model agents at multi-step security tasks, and which steps remain difficult when intermediate goals are supplied?
Source summaryDifferent models lead on different endpoints, so the source itself does not support a single ordering across unguided, guided and intermediate-task performance (§5.1).
NUWA research note · independent review pending · Contains: question / method / limits
CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale
QuestionCan agents reproduce real historical software vulnerabilities at scale, and how do input information, model and agent configuration change that ability?
Source summaryThe difficulty ladder demonstrates that richer diagnostic information makes reproduction easier for the examined configuration (§4, Figure 6).
NUWA research note · independent review pending · Contains: question / method / limits
Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios
QuestionHow do model generations and inference budgets affect progress on linked cyber-range tasks?
Source summaryAt 10M tokens, mean corporate-range progress rose from 1.7 steps for GPT-4o to 9.8 for Opus 4.6.
NUWA research note · independent review pending · Contains: question / method / interpretation / limits
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
QuestionCan tool-using agents maintain utility while processing attacker-controlled external data?
Source summaryAgents fail some benign tasks even without attacks.
NUWA research note · independent review pending · Contains: question / method / limits
SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?
QuestionCan agents reproduce specific disclosed vulnerabilities in large security-critical codebases?
Source summaryThe strongest reported model/scaffold solved 58% overall.
NUWA research note · independent review pending · Contains: question / method / interpretation / limits
Evidence matrix
Rows are research questions; columns show what published evaluations observe. Counts refer to evaluation dossiers, not risk levels.
| Risk questions | Propensity | Capability | Behavior |
|---|---|---|---|
| Cyber task executionCan an AI system complete multi-step cyber tasks in authorized test environments? | —No evidence in the current catalogue | 15evaluation dossiers | —No evidence in the current catalogue |
| Vulnerability reproductionCan an agent reproduce a documented vulnerability under test conditions? | —No evidence in the current catalogue | 2evaluation dossiers | —No evidence in the current catalogue |
| Agent compromise through untrusted inputHow do untrusted tool outputs redirect an agent away from the authorized task? | —No evidence in the current catalogue | —No evidence in the current catalogue | 2evaluation dossiers |
The paths show research relationships, not proven causal chains or a risk score.
Cybersecurity / Propensity
Cyber task execution
Can an AI system complete multi-step cyber tasks in authorized test environments?
- Included
- Range and CTF task success with access and budget disclosed.
- Does not establish
- Success in a test is not an observed real-world attack.
No evidence in the current catalogue
Cybersecurity / Capability
Cyber task execution
Can an AI system complete multi-step cyber tasks in authorized test environments?
- Included
- Range and CTF task success with access and budget disclosed.
- Does not establish
- Success in a test is not an observed real-world attack.
15 evaluation dossiers · 116 verified result records
AgentCyberRange ↗
The source directly measures the stated construct under its protocol.
Assessment subject: AI agent completing cyber-range tasks
Evidence relationship: Official leaderboard, Level-0 tracks
Cybench ↗
The source directly measures the stated construct under its protocol.
Assessment subject: AI agent solving CTF tasks
Evidence relationship: Paper abstract and task protocol
Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios ↗
Task completion in a bounded range measures capability under its specific protocol.
Assessment subject: AI agent systems
Evidence relationship: Research abstract; full technical paper linked
SEC-bench Pro ↗
Vulnerability reproduction informs cyber task capability but not a full intrusion chain.
Assessment subject: Coding agents on security tasks
Evidence relationship: arXiv:2605.26548v2, task definition
SandboxEscapeBench ↗
The benchmark directly tests adversarial sandbox-escape tasks.
Assessment subject: Agents with container shell access
Evidence relationship: AISI abstract, nested sandbox and threat model
EVMbench ↗
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
Assessment subject: The model or agent in the stated task configuration.
Evidence relationship: Original protocol method and limitations
CTF-Archive-Diamond ↗
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
Assessment subject: The model or agent in the stated task configuration.
Evidence relationship: Original protocol method and limitations
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/3/on_target
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/3/flag_captured
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/6/on_target
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/11/on_target
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/11/subRows/0/on_target
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/12/on_target
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/14/on_target
ExploitGym · ExploitGym official snapshot ↗
The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.
Assessment subject: Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
Evidence relationship: /results/14/subRows/0/on_target
Cybersecurity / Behavior
Cyber task execution
Can an AI system complete multi-step cyber tasks in authorized test environments?
- Included
- Range and CTF task success with access and budget disclosed.
- Does not establish
- Success in a test is not an observed real-world attack.
No evidence in the current catalogue
Cybersecurity / Propensity
Vulnerability reproduction
Can an agent reproduce a documented vulnerability under test conditions?
- Included
- Validated reproduction against historical targets.
- Does not establish
- Reproduction is not discovery of a new vulnerability.
No evidence in the current catalogue
Cybersecurity / Capability
Vulnerability reproduction
Can an agent reproduce a documented vulnerability under test conditions?
- Included
- Validated reproduction against historical targets.
- Does not establish
- Reproduction is not discovery of a new vulnerability.
2 evaluation dossiers · 154 verified result records
CyberGym ↗
The source directly measures the stated construct under its protocol.
Assessment subject: AI agent reproducing historical vulnerabilities
Evidence relationship: Paper abstract and validated trigger protocol
SEC-bench Pro ↗
The paper evaluates working reproductions of disclosed bugs.
Assessment subject: Coding agents with repository and report context
Evidence relationship: arXiv:2605.26548v2, abstract and benchmark construction
Cybersecurity / Behavior
Vulnerability reproduction
Can an agent reproduce a documented vulnerability under test conditions?
- Included
- Validated reproduction against historical targets.
- Does not establish
- Reproduction is not discovery of a new vulnerability.
No evidence in the current catalogue
Cybersecurity / Propensity
Agent compromise through untrusted input
How do untrusted tool outputs redirect an agent away from the authorized task?
- Included
- Prompt-injection trials with attacker objective and benign-task utility.
- Does not establish
- This tests exposed agents, not the agent model’s desire to attack.
No evidence in the current catalogue
Cybersecurity / Capability
Agent compromise through untrusted input
How do untrusted tool outputs redirect an agent away from the authorized task?
- Included
- Prompt-injection trials with attacker objective and benign-task utility.
- Does not establish
- This tests exposed agents, not the agent model’s desire to attack.
No evidence in the current catalogue
Cybersecurity / Behavior
Agent compromise through untrusted input
How do untrusted tool outputs redirect an agent away from the authorized task?
- Included
- Prompt-injection trials with attacker objective and benign-task utility.
- Does not establish
- This tests exposed agents, not the agent model’s desire to attack.
2 evaluation dossiers · 19 verified result records
AgentDojo ↗
The source directly measures the stated construct under its protocol.
Assessment subject: Defender agent under injected tool output
Evidence relationship: AgentDojo v3, attack success and utility metrics
TraceProbe ↗
Editorial scope association; heterogeneous scores are not equivalent.
Assessment subject: Can execution traces identify safety violations that final outcomes miss?
Evidence relationship: §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483
References
- [1] Evaluating Kimi K3 for Misalignment Risks. 2026-09-11. September 2026 ; §2.1.1; p. 19; §2.1.2; p. 19
- [2] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; §1.1; §1.2
- [3] GLM-5.2 Risk Evaluation Report. 2026-08-02. v1.1 · revised 26 August 2026 ; §2.1; Appendix A.1; §2.2; Appendix A.2
- [4] Evaluation of DeepSeek AI Models. 2025-09-30. 30 September 2025 ; §4.1.2; pp.27–28; §5.1
- [5] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026 ; Capability Results; Figure 2; Original protocol method and limitations
- [6] Claude Opus 4.5 System Card. 2025-11-05. 2025-12-05 ; PDF table of contents → named RSP / cyber / alignment sections
- [7] Claude Opus 4.6 System Card. 2026-02-06. 2026-03-06 ; PDF contents → named RSP, agentic safety and alignment sections
- [8] Claude Sonnet 4.6 System Card. 2026-02-17. 2026-03-06 ; PDF table of contents → named RSP / cyber / alignment sections
- [9] Claude Opus 4.7 System Card. 2026-04-16 ; PDF table of contents → named RSP / cyber / alignment sections
- [10] Claude Opus 4.8 System Card. 2026-05-28. 2026-06-17 ; PDF table of contents → named RSP / cyber / alignment sections
- [11] Claude Sonnet 5 System Card. 2026-06-30. 2026-07-10 ; PDF table of contents → named RSP / cyber / alignment sections
- [12] Muse Wonder Model Card. 2026-10-23 ; PDF pp.7–10
- [13] Muse Spark Safety & Preparedness Report. 2026-05-14. 2026-08-24 ; https://arxiv.org/html/2606.12429#S2.SS2; https://arxiv.org/html/2606.12429#S3.SS2
- [14] Muse Spark 1.1 Evaluation Report. 2026-07-09 ; PDF §§2.1–2.3; PDF §3
- [15] GPT-5.6 System Card. 2026-07-09 ; Original HTML → named Preparedness / Alignment / CoT Monitorability sections
- [16] GPT-6 Sol and GPT-6 Luna: October 2026 update. 2026-10-07 ; https://deploymentsafety.openai.com/gpt-6-october#section.8
- [17] Grok 4.7 Model Card. 2026-09-21 ; PDF pp.17–19 (printed 16–18)
- [18] CyberGym official leaderboard and machine-data extraction ; /level1/0/score_10; /level1/64/score_10
- [19] ExploitGym official leaderboard and benchmark page ; /results/3/on_target; /results/3/flag_captured; /results/6/on_target
- [20] SEC-bench Pro official leaderboard: selected 260617 and historical 260505 ; /0/results/0/score_modes/headline/score; /0/results/0/score_modes/completed/score; /1/results/0/score_modes/headline/score
- [21] AgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges ; routes-B-0Mubrq.js::QR/1/level0/pass3; routes-B-0Mubrq.js::$R/1/level0/pass3; Official leaderboard, Level-0 tracks
- [22] Andy K. Zhang, Neil Perry, et al.. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. 2025-04-12. v4 ; Paper abstract and task protocol
- [23] Linus Folkerts, Will Payne, et al.. Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios. Research page checked 2026-10-25 ; Research abstract; full technical paper linked
- [24] Zhun Wang, Tianneng Shi, et al.. CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale. 2026-03-24. v3 ; Paper abstract and validated trigger protocol
- [25] Edoardo Debenedetti, Jie Zhang, et al.. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. 2024-11-24. v3 ; AgentDojo v3, attack success and utility metrics
- [26] Hwiwon Lee ∗ Jiawei Liu Dongjun Kim Wubing Xia Ziqi Zhang Chunqiu Steven Xia Lingming Zhang ∗. SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?. 2026-07-20. v2 ; arXiv:2605.26548v2, abstract and benchmark construction; arXiv:2605.26548v2, task definition
- [27] Rahul Marchand, Art O Cathain, et al.. Quantifying Frontier LLM Capabilities for Container Sandbox Escape. AISI research page checked 2026-10-25 ; AISI abstract, nested sandbox and threat model
- [28] EVMbench ; Original protocol method and limitations
- [29] TraceProbe: An Execution-Based Benchmark for Diagnosing Safety Failures in LLM Agents ; §3.6; §5–6; Tables 2,15,17; limitations; text lines 00483