NUWA Evaluation dossier
ExploitGym
Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
ExploitGym · Intended-target successes
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
20 observations · 10 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
- v1
- v0
- Version not specified
10 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
20 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 GPT-5.6 Sol (reasoning max) · Codex CLI | 293 countIntended-target successes | Incomplete | v1on_target6h timeoutCodex CLI | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
 Claude Mythos 5 | 247 countIntended-target successes | Incomplete | Version not specifiedon_target6h timeout— | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| GPT-5.6 Sol (reasoning max) · Codex CLI | 216 countIntended-target successes | Incomplete | v1on_target2h timeout cutoffCodex CLI | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| Claude Opus 5 | 191 countIntended-target successes | Incomplete | Version not specifiedon_target6h timeout— | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| Claude Mythos 5 | 181 countIntended-target successes | Incomplete | Version not specifiedon_target2h timeout cutoff— | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| Claude Opus 5 | 171 countIntended-target successes | Incomplete | Version not specifiedon_target2h timeout cutoff— | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
 GLM-5.3 · Claude Code | 130 countIntended-target successes | Incomplete | Version not specifiedon_targetrescaled 6h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| GPT-5.5 (OpenAI's default safety filters block all GPT-5.5 exploit attempts under default prompting.) · Codex CLIGPT-5.5 | 129 countIntended-target successes | Incomplete | v1on_target2h timeoutCodex CLI | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-04-23Report — |
| Claude Opus 4.8 | 120 countIntended-target successes | Incomplete | Version not specifiedon_target6h timeout— | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-05-28Report — |
| GLM-5.3 · Claude Code | 105 countIntended-target successes | Incomplete | Version not specifiedon_targetrescaled 2h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| Claude Opus 4.8 | 80 countIntended-target successes | Incomplete | Version not specifiedon_target2h timeout cutoff— | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-05-28Report — |
| GLM-5.2 · DoGNAVYGLM-5.2 | 79 countIntended-target successes | Incomplete | Version not specifiedon_targetselected subsetDoGNAVY | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-06-16Report — |
| GPT-5.4 · Codex CLIGPT-5.4 | 61 countIntended-target successes | Incomplete | v1on_target2h timeoutCodex CLI | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-03-05Report — |
| GLM-5.2 · Claude CodeGLM-5.2 | 39 countIntended-target successes | Incomplete | Version not specifiedon_targetrescaled 6h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-06-16Report — |
| GLM-5.2 · Claude CodeGLM-5.2 | 29 countIntended-target successes | Incomplete | Version not specifiedon_targetrescaled 2h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-06-16Report — |
| Claude Opus 4.6 (Results obtained in collaboration with Anthropic.) · Claude CodeClaude Opus 4.6 | 16 countIntended-target successes | Incomplete | v1on_target2h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-02-05Report — |
| Claude Opus 4.7 · Claude CodeClaude Opus 4.7 | 12 countIntended-target successes | Incomplete | v1on_target2h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-04-16Report — |
 Gemini 3.1 Pro · Gemini CLIGemini 3.1 Pro | 12 countIntended-target successes | Incomplete | v0on_target2h timeoutGemini CLI | Not disclosedExploitGym official leaderboard and benchmark page | Release 2026-02-19Report — |
 Muse Spark 1.1 (helpful-only versoin) · Meta Agent (off-the-shelf agent harness) | 7 countIntended-target successes | Incomplete | v1on_target4h timeoutMeta Agent (off-the-shelf agent harness) | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
| GLM-5.1 · Claude Code | 4 countIntended-target successes | Incomplete | v1on_target2h timeoutClaude Code | Not disclosedExploitGym official leaderboard and benchmark page | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Evaluates exploit construction across real software targets and checks whether success uses the designated vulnerability.
- Scoring / outcome
- Intended-target successes · Source-reported measurement under the recorded configuration.
- Interpretation boundary
- Target mitigations, supplied artifacts and vulnerability attribution must accompany any performance claim.
A reported evaluation
The source evaluates GPT-5.6 Sol (reasoning max) · Codex CLI · v1.
Run configuration: Codex CLI · 6h timeout.
Reported measurement: Intended-target successes · 293 count.
Source-reported measurement under the recorded configuration. Per-result sample size not disclosed
Intended-target successes: 293 count. Conditions are retained from the source.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can agents extend supplied vulnerability triggers into concrete impact across systems?
- Task population
- 898 instances span userspace, V8 and Linux, with reproducible containers, varied protections and a time/cost-gated attempt.
- Scoring and failure modes
- Raw flag success and intended-vulnerability alignment are separate outcomes.
- System conditions
- 898 instances; userspace/V8/Linux protection variants and time/cost-gated attempts.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: GPT-5.5 and Mythos Preview aligned on only 56.7% and 69.5% of their flagged solves (§4).
- Validity and interpretation
- Ground-truth exploits are unavailable for every task; refusal, tool failure and impossible tasks can all lower observed success (§5).
- Appropriate research use
- Model recommendation: retain alignment-audit results alongside raw success, protection setting and attempt budget.
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · AbstractSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · §3 ExploitGym BenchmarkSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · §4 EvaluationSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
- ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks? · §5 Discussion and ConclusionSelected original resource https://arxiv.org/html/2605.11086; sha256:06690e18ab2d762801ec3f66a49531df74744a18f15e2bc9e54866e8e3c8e23e; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
ExploitGym website
ExploitGym official leaderboard and benchmark page →
Actual evaluators: Not disclosed
21 result records · 14 reported system names
Protocols and comparison decisions (12)
ExploitGym: Flag captures · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
ExploitGym: Intended-target successes · ExploitGym websitesource-report · Scope reviewed
Original source charts and materials
Evaluation results
Filter systems and configurations
Reported by: ExploitGym website · Executed by: Not established for every submission · Captured: 2026-10-26
Intended-target success and flag capture are distinct counts. Task versions and timeout cutoffs remain separate. Catalogue size is not assumed to be the denominator for each submission.
8 of 8 systems shown in source order
Intended-target successes · v1
- Codex CLI · 6h timeout
- Codex CLI · 2h timeout cutoff
- Meta Agent (off-the-shelf agent harness) · 4h timeout
- Codex CLI · 2h timeout
- GPT-5.4 · Codex CLI61 countCodex CLI · 2h timeout
- Claude Code · 2h timeout
- Claude Opus 4.7 · Claude Code12 countClaude Code · 2h timeout
- GLM-5.1 · Claude Code4 countClaude Code · 2h timeout
| System | Result | Task set / mode | Configuration | Dates / attribution |
|---|---|---|---|---|
| GPT-5.6 Sol (reasoning max) · Codex CLI | 293 count | v1on_target | Codex CLI6h timeout · ? trials | 2026-07-13Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GPT-5.6 Sol (reasoning max) · Codex CLI | 216 count | v1on_target | Codex CLI2h timeout cutoff · ? trials | Not specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Muse Spark 1.1 (helpful-only versoin) · Meta Agent (off-the-shelf agent harness) | 7 count | v1on_target | Meta Agent (off-the-shelf agent harness)4h timeout · ? trials | 2026-07-09Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GPT-5.5 (OpenAI's default safety filters block all GPT-5.5 exploit attempts under default prompting.) · Codex CLI | 129 count | v1on_target | Codex CLI2h timeout · ? trials | 2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GPT-5.5 (OpenAI's default safety filters block all GPT-5.5 exploit attempts under default prompting.) · Codex CLI | 208 count | v1flag_captured | Codex CLI2h timeout · ? trials | 2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GPT-5.4 · Codex CLI | 61 count | v1on_target | Codex CLI2h timeout · ? trials | 2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Opus 4.6 (Results obtained in collaboration with Anthropic.) · Claude Code | 16 count | v1on_target | Claude Code2h timeout · ? trials | 2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Gemini 3.1 Pro · Gemini CLI | 12 count | v0on_target | Gemini CLI2h timeout · ? trials | 2026-05-13Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Opus 4.7 · Claude Code | 12 count | v1on_target | Claude Code2h timeout · ? trials | 2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GLM-5.1 · Claude Code | 4 count | v1on_target | Claude Code2h timeout · ? trials | 2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Mythos 5 | 247 count | Version not specifiedon_target | Not specified6h timeout · ? trials | 2026-06-09Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Mythos 5 | 181 count | Version not specifiedon_target | Not specified2h timeout cutoff · ? trials | Not specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Opus 5 | 191 count | Version not specifiedon_target | Not specified6h timeout · ? trials | 2026-07-24Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Opus 5 | 171 count | Version not specifiedon_target | Not specified2h timeout cutoff · ? trials | Not specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Opus 4.8 | 120 count | Version not specifiedon_target | Not specified6h timeout · ? trials | 2026-05-28Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| Claude Opus 4.8 | 80 count | Version not specifiedon_target | Not specified2h timeout cutoff · ? trials | Not specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GLM-5.2 · DoGNAVY | 79 count | Version not specifiedon_target | DoGNAVYselected subset · ? trials | 2026-08-22Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GLM-5.3 · Claude Code | 130 count | Version not specifiedon_target | Claude Coderescaled 6h timeout · ? trials | 2026-08-18Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GLM-5.3 · Claude Code | 105 count | Version not specifiedon_target | Claude Coderescaled 2h timeout · ? trials | Not specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GLM-5.2 · Claude Code | 39 count | Version not specifiedon_target | Claude Coderescaled 6h timeout · ? trials | 2026-08-18Source row date; semantics unspecified · ExploitGym website · Executor: Not specified |
| GLM-5.2 · Claude Code | 29 count | Version not specifiedon_target | Claude Coderescaled 2h timeout · ? trials | Not specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified |
Assessment provenance
ExploitGym official snapshot → · External report
Related research questions
References
- [1] Z.ai release notes ; GLM-5.2 entry dated 2026-06-16
- [2] GLM-5.2 model card ; Parameter counts not disclosed in reviewed developer source
- [3] OpenAI — introducing gpt 5 5 ; Publication date and availability section; Parameter counts not disclosed in launch article
- [4] Introducing Claude Opus 4.7 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [5] OpenAI — introducing gpt 5 4 ; Publication date and availability section; Parameter counts not disclosed in launch article
- [6] Claude Opus 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [7] Gemini 3.1 Pro: Announcing our latest Gemini AI model ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [8] Introducing Claude Opus 4.8 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [9] ExploitGym official leaderboard and benchmark page ; /results/0/on_target; /results/0/subRows/0/on_target; /results/1/on_target
- [10] Zhun Wang, Nico Schiller, et al.. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?. 2026-05-13
- [11] Zhun Wang, Nico Schiller, et al.. ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?. 2026-05-13 ; Author affiliations on the original paper title page; Figure 1