Evaluation dossier
AgentCyberRange
Autonomous cyber tasks in enterprise-like ranges.
Research question
AgentCyberRange evaluates autonomous cyber tasks in enterprise-like ranges. Its Web exploitation and post-exploitation tracks expose different parts of the task, allowing readers to distinguish an initial objective from continued action after an established starting point.
Representative workflow: two tracks
In the Web track, a submission is assessed on the Web exploitation task defined by the project. In the post-exploitation track, assessment concerns the subsequent range objective. These are source-defined task families, not two interchangeable scores for a single event.
The useful question is where a system succeeds or stops along a longer task. The project’s track definitions provide that context; a leaderboard percentage alone does not describe the whole trajectory.
Task and scoring
NUWA preserves the project snapshot, submitted model and harness, track, metric and task level. Each submission’s conditions remain attached to its observation. The result charts read those stored values rather than numbers transcribed into this article.
Reading the results
Keep the two tracks separate. Changes in task sets, model endpoints and submitted agent configurations can change the meaning of a comparison. An improvement within a fixed configuration answers a different question from a change in the complete agent system.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
AgentCyberRange · Web exploitation · Pass@3
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
22 observations · 6 dated
- Website's Level-0 web and post-exploitation tracks
- Web exploitation
16 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
22 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
| 35.37%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksCodex 0.144.1 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 35.37%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Codex | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 26.67%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksCodex 0.147.0 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 26.67%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Codex | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release 2026-04-23Report — | |
| 24.61%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksKimi Code 0.35.0 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 24.61%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Kimi Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release 2026-07-16Report — | |
| 22.39%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksQwen Code 0.21.10 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 22.39%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Qwen Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 18.97%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksCodex 0.133.0 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 18.97%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Codex | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release 2026-04-23Report — | |
| 14.19%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksClaude Code 2.1.226 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 14.19%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Claude Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release 2026-04-24Report — | |
| 14.02%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksClaude Code 2.1.150 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 14.02%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Claude Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release 2026-04-16Report — | |
| 13.51%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksQwen Code 0.16.1 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 13.51%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Qwen Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 11.97%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksClaude Code 2.1.150 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 11.97%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Claude Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 10.26%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksClaude Code 2.1.150 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 10.26%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Claude Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release 2026-04-24Report — | |
| 5.81%Web · Level-0 Pass@3 | Incomplete | Website's Level-0 web and post-exploitation tracksKimi Code 1.44.0 | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — | |
| 5.81%Success rate (Pass@3) | Incomplete | Web exploitationlevel0Kimi Code | Not disclosedAgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- AI agent completing cyber-range tasks
- Measurement target
- Task success across web exploitation and post-exploitation settings; configurations must be checked per submission.
- Scoring / outcome
- Web · Level-0 Pass@3 · The website's reported Level-0 Pass@3 on web tasks; retain its metric definition.
- Interpretation boundary
- Changing task sets and submitted agent configurations prevent unrestricted comparison.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can an agent discover an attack surface from sparse entry information and continue making decisions as access and information accumulate?
- Task population
- The team introduction starts agents from entry IP information and describes progression through linked environments. Earlier access and discovered clues change later options. Web and post-access tracks are presented separately. The trajectory figure retains unsuccessful exploration as well as successful progress, illustrating a process rather than just a final endpoint.
- Scoring and failure modes
- The existing dated catalogue preserves the official site’s Level-0 Pass@3 labels. The fetched introduction does not independently establish the pass@3 estimator, per-cell trial denominators, complete budgets or all submission settings. Those fields remain unknown in this new profile.
- System conditions
- The team introduction starts agents from entry IP information and describes progression through linked environments. Earlier access and discovered clues change later options. Web and post-access tracks are presented separately. The trajectory figure retains unsuccessful exploration as well as successful progress, illustrating a process rather than just a final endpoint.
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- The cached project page is a JavaScript shell. The accessible research introduction omits complete scoring, budget, model-release identity and independent-run counts. A trajectory figure cannot establish a population rate. The introduction’s application requests do not prove deployments or partnerships.
- Appropriate research use
- Model recommendation: obtain the archived benchmark specification and submission configuration before interpreting model gaps; use the introduction for task scope and process reading only.
- AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks · 研究问题Research introduction captured 2026-10-27T09:55:41.654036+00:00; sha256:82b125ff6ee2fc02acde66bd7e5debed251334d1db9d88f7831aaae13e267a7f
- AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks · 长程攻击轨迹:状态与线索复用;失败路径与任务过程;图 2Research introduction captured 2026-10-27T09:55:41.654036+00:00; sha256:82b125ff6ee2fc02acde66bd7e5debed251334d1db9d88f7831aaae13e267a7f
- AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks · 任务设置与环境:攻击链与状态依赖;环境规模Research introduction captured 2026-10-27T09:55:41.654036+00:00; sha256:82b125ff6ee2fc02acde66bd7e5debed251334d1db9d88f7831aaae13e267a7f
- AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks · 适用范围Research introduction captured 2026-10-27T09:55:41.654036+00:00; sha256:82b125ff6ee2fc02acde66bd7e5debed251334d1db9d88f7831aaae13e267a7f
- AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks · 项目与更新Research introduction captured 2026-10-27T09:55:41.654036+00:00; sha256:82b125ff6ee2fc02acde66bd7e5debed251334d1db9d88f7831aaae13e267a7f
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
AgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges
AgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges
Actual evaluators: Not disclosed
11 result records · 11 reported system names
Protocols and comparison decisions (2)
AgentCyberRange website
AgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges →
Actual evaluators: Not disclosed
22 result records · 11 reported system names
Protocols and comparison decisions (8)
Original source charts and materials
Evaluation results
Filter systems and configurations
Reported by: AgentCyberRange website · Executed by: Not established for every submission · Captured: 2026-10-26
Web exploitation and post-exploitation are separate evaluations. Pass@3 retains the site’s definition; row-specific budgets, sample counts and execution dates are not disclosed.
11 of 11 systems shown in source order
Success rate (Pass@3) · Web exploitation · level0
- Codex · Budget not specified
GPT-5.5 / Codex 0.133.018.97%Codex · Budget not specified
GPT-5.5 / Codex 0.147.026.67%Codex · Budget not specified- Claude Code · Budget not specified
- Claude Code · Budget not specified
- Claude Code · Budget not specified
- Claude Code · Budget not specified
- Qwen Code · Budget not specified
- Qwen Code · Budget not specified
- Kimi Code · Budget not specified
- Kimi Code · Budget not specified
| System | Result | Task set / mode | Configuration | Dates / attribution |
|---|---|---|---|---|
| 35.37% | Web exploitationlevel0 | CodexNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 18.97% | Web exploitationlevel0 | CodexNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 26.67% | Web exploitationlevel0 | CodexNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 14.02% | Web exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 11.97% | Web exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 10.26% | Web exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 14.19% | Web exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 13.51% | Web exploitationlevel0 | Qwen CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 22.39% | Web exploitationlevel0 | Qwen CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 5.81% | Web exploitationlevel0 | Kimi CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 24.61% | Web exploitationlevel0 | Kimi CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 74.1% | Post exploitationlevel0 | CodexNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 31.71% | Post exploitationlevel0 | CodexNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 42.07% | Post exploitationlevel0 | CodexNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 15.04% | Post exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 11.37% | Post exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 12.2% | Post exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 28.46% | Post exploitationlevel0 | Claude CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 13.02% | Post exploitationlevel0 | Qwen CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 27.64% | Post exploitationlevel0 | Qwen CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 5.68% | Post exploitationlevel0 | Kimi CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified | |
| 43.9% | Post exploitationlevel0 | Kimi CodeNot specified · ? trials | Not specifiedSource row date; semantics unspecified · AgentCyberRange website · Executor: Not specified |
Assessment provenance
AgentCyberRange official snapshot → · External report
Related research questions
Cybersecurity / Cyber task execution →
The source directly measures the stated construct under its protocol.
References
- [1] AgentCyberRange: Evaluate Agent Cyberattack Capabilities in Enterprise-like Cyber Ranges ; Official project: Web exploitation / Post exploitation tracks; Level-0 submission records; Official leaderboard: Level-0 Pass@3; arrays QR (web), $R (post), version map ez; routes-B-0Mubrq.js::QR/0/level0/pass3
- [2] Kimi Code — What’s New ; Kimi K3 · July 16, 2026
- [3] kimi-k3 developer model card ; Model Summary: Total Parameters; Activated Parameters
- [4] deepseek-v4-pro release ; News article title, dated heading and availability
- [5] deepseek-v4-pro developer model card ; Introduction; model comparison table
- [6] OpenAI — introducing gpt 5 5 ; Publication date and availability section; Parameter counts not disclosed in launch article
- [7] Introducing Claude Opus 4.7 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [8] deepseek-v4-flash developer model card ; Introduction; model comparison table
- [9] AgentCyberRange: Evaluating autonomous cyber capabilities over long-horizon tasks ; Research introduction: Fudan–Whitzard team