WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← NUWA Index

Source record

ExploitGym

Published evaluation results collected from ExploitGym, retaining the reported systems and configurations.

Original source and provenance ↓

Evaluation results

Reported by: ExploitGym website · Executed by: Not established for every submission · Captured: 2026-10-26

Intended-target success and flag capture are distinct counts. Task versions and timeout cutoffs remain separate. Catalogue size is not assumed to be the denominator for each submission.

8 of 8 systems shown in source order

Intended-target successes · v1

ExploitGymIntended-target successes · v1Source order · native units · configuration-specific082164246328GPT-5.6 Sol (reasoning max) · Codex CLI: 293 count; v1; 6h timeoutGPT-5.6 Sol (reasoning max) · CodexCLI293 countGPT-5.6 Sol (reasoning max) · Codex CLI: 216 count; v1; 2h timeout cutoffGPT-5.6 Sol (reasoning max) · CodexCLI216 countMuse Spark 1.1 (helpful-only versoin) · Meta Agent (off-the-shelf agent harness): 7 count; v1; 4h timeoutMuse Spark 1.1 (helpful-only versoin)· Meta Agent (off-the-shelf agentharness)7 countGPT-5.5 (OpenAI's default safety filters block all GPT-5.5 exploit attempts under default prompting.) · Codex CLI: 129 count; v1; 2h timeoutGPT-5.5 (OpenAI's default safetyfilters block all GPT-5.5 exploitattempts under default prompting.) ·Codex CLI129 countGPT-5.4 · Codex CLI: 61 count; v1; 2h timeoutGPT-5.4 · Codex CLI61 countClaude Opus 4.6 (Results obtained in collaboration with Anthropic.) · Claude Code: 16 count; v1; 2h timeoutClaude Opus 4.6 (Results obtained incollaboration with Anthropic.) ·Claude Code16 countClaude Opus 4.7 · Claude Code: 12 count; v1; 2h timeoutClaude Opus 4.7 · Claude Code12 countGLM-5.1 · Claude Code: 4 count; v1; 2h timeoutGLM-5.1 · Claude Code4 counthttps://www.cybergym.io/exploitgym/
  1. Codex CLI · 6h timeout
  2. Codex CLI · 2h timeout cutoff
  3. Meta Agent (off-the-shelf agent harness) · 4h timeout
  4. Codex CLI · 2h timeout
  5. Codex CLI · 2h timeout
  6. Claude Code · 2h timeout
  7. Claude Code · 2h timeout
  8. Claude Code · 2h timeout
Source-reported systems and evaluation conditions
SystemResultTask set / modeConfigurationDates / attribution
GPT-5.6 Sol (reasoning max) · Codex CLI293 countv1on_targetCodex CLI6h timeout · ? trials2026-07-13Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GPT-5.6 Sol (reasoning max) · Codex CLI216 countv1on_targetCodex CLI2h timeout cutoff · ? trialsNot specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified
Muse Spark 1.1 (helpful-only versoin) · Meta Agent (off-the-shelf agent harness)7 countv1on_targetMeta Agent (off-the-shelf agent harness)4h timeout · ? trials2026-07-09Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GPT-5.5 (OpenAI's default safety filters block all GPT-5.5 exploit attempts under default prompting.) · Codex CLI129 countv1on_targetCodex CLI2h timeout · ? trials2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GPT-5.5 (OpenAI's default safety filters block all GPT-5.5 exploit attempts under default prompting.) · Codex CLI208 countv1flag_capturedCodex CLI2h timeout · ? trials2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GPT-5.4 · Codex CLI61 countv1on_targetCodex CLI2h timeout · ? trials2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Opus 4.6 (Results obtained in collaboration with Anthropic.) · Claude Code16 countv1on_targetClaude Code2h timeout · ? trials2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Gemini 3.1 Pro · Gemini CLI12 countv0on_targetGemini CLI2h timeout · ? trials2026-05-13Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Opus 4.7 · Claude Code12 countv1on_targetClaude Code2h timeout · ? trials2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GLM-5.1 · Claude Code4 countv1on_targetClaude Code2h timeout · ? trials2026-06-16Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Mythos 5247 countVersion not specifiedon_targetNot specified6h timeout · ? trials2026-06-09Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Mythos 5181 countVersion not specifiedon_targetNot specified2h timeout cutoff · ? trialsNot specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Opus 5191 countVersion not specifiedon_targetNot specified6h timeout · ? trials2026-07-24Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Opus 5171 countVersion not specifiedon_targetNot specified2h timeout cutoff · ? trialsNot specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Opus 4.8120 countVersion not specifiedon_targetNot specified6h timeout · ? trials2026-05-28Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
Claude Opus 4.880 countVersion not specifiedon_targetNot specified2h timeout cutoff · ? trialsNot specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified
GLM-5.2 · DoGNAVY79 countVersion not specifiedon_targetDoGNAVYselected subset · ? trials2026-08-22Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GLM-5.3 · Claude Code130 countVersion not specifiedon_targetClaude Coderescaled 6h timeout · ? trials2026-08-18Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GLM-5.3 · Claude Code105 countVersion not specifiedon_targetClaude Coderescaled 2h timeout · ? trialsNot specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified
GLM-5.2 · Claude Code39 countVersion not specifiedon_targetClaude Coderescaled 6h timeout · ? trials2026-08-18Source row date; semantics unspecified · ExploitGym website · Executor: Not specified
GLM-5.2 · Claude Code29 countVersion not specifiedon_targetClaude Coderescaled 2h timeout · ? trialsNot specifiedSource row date; semantics unspecified · ExploitGym website · Executor: Not specified

Scope and attribution

Reported by
ExploitGym website
Actual evaluator
Not established for every observation
Captured version
2026-10-26
Source identity
8c8a725ba1775411

Intended-target success and flag capture are distinct counts. Task versions and timeout cutoffs remain separate. Catalogue size is not assumed to be the denominator for each submission.

Source findings

The following statements summarize the source. They are not independent NUWA experimental findings.

Leaderboard success requires the intended vulnerability; flag capture alone is a separate outcome.

exploitgym.html:123-127,212-214

Scope is limited to the source conditions.

The page says evaluation used trusted security-research access programs.

exploitgym.html:124-125

Scope is limited to the source conditions.

The page reports that mitigations reduced successes but did not eliminate all of them.

exploitgym.html:149-153

Scope is limited to the source conditions.

The authors frame automated exploitation evaluation as dual-use research requiring responsible measurement.

exploitgym.html:157-161

Scope is limited to the source conditions.

The narrative argues that stronger agents can benefit from longer budgets; its study conditions are incompletely specified.

exploitgym.html:262-266

Scope is limited to the source conditions.

Academic authors developed the benchmark methodology; industry partners supplied model access and feedback.

exploitgym.html:350-353

Scope is limited to the source conditions.

Extraction coverage

Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.

View 14 coverage entries
SectionTreatmentLocation and scope
Title, authors, affiliations and citationqualitativeexploitgym.html:37-77,340-353Identity and methodological attribution retained; page provides citation year 2026 but no precise page publication or revision date.
Benchmark scope and task inventoryextractedhttps://www.cybergym.io/assets/data/exploitgym.json#/instances; exploitgym.html:169-214All four exact inventory counts retained as catalogue metadata, not row denominators.
All primary submission fieldsextractedhttps://www.cybergym.io/assets/data/exploitgym.json#/results/0..1415 primary objects: 14 visible and 1 hidden. Every reported score and usage leaf, including explicit nulls, retained.
All six nested budget variantsextractedhttps://www.cybergym.io/assets/data/exploitgym.json#/results/{0,9,10,11,13,14}/subRows/0Separate contexts; parent inheritance is explicit. 2h cutoff, 4h, 6h, rescaled and selected-subset conditions stay distinct.
Visibility, rank metric, version display, averages and inheritanceextractedexploitgym.html:360-513; leaderboard.js:225-238hidden rows filtered; total prefers on_target; inherited parent metadata preserved; average presentation rounding not extracted as new values.
Key Takeawaysqualitativeexploitgym.html:133-163Only high-level conclusions retained; operational examples excluded.
Flag capture versus intended vulnerabilityextractedexploitgym.html:229-239Narrative numbers separately quarantined; GPT-5.5 conflict and hidden Mythos Preview recorded.
Different models find different exploitsextractedexploitgym.html:243-254Five explicit overlap counts retained with unresolved population; no figure estimates or inferred model totals.
More budget helpsextractedexploitgym.html:257-267Two exact Mythos narrative endpoints, approximate Opus plateau and stated time bound retained as unresolved narrative evidence.
Overview, flag, overlap and time-curve imagesfigure-onlyexploitgym.html:192,231,253,260Image files not fetched/read; no bar-height or curve estimates. Exact adjacent prose values extracted.
Example vulnerability and exploit trajectorynot-relevantexploitgym.html:274-309Operational exploit/task details and incidental example-specific numbers deliberately excluded from extraction.
Why This Mattersqualitativeexploitgym.html:311-333High-level defense and responsible-evaluation implications retained.
External system cards, reports and submissionsreusedhttps://www.cybergym.io/assets/data/exploitgym.json#/results/*/source_urlSource links preserved as dependencies; none independently read or counted as replication in this unit.
Exact executor, trials, row denominator, date semantics and missing fieldsnot-disclosedhttps://www.cybergym.io/assets/data/exploitgym.json#/resultsNo inference from model brand, reporting party, dates, absent metrics or catalogue size.

Unresolved source issues

正文和结构化榜单部分统计口径未完全对应，保留各自范围，不据此合并或修正数值。

部分提交未披露版本、样本分母或执行方；重标定预算条件与普通时限条件分别保留。

Version fingerprints
document

https://www.cybergym.io/exploitgym/

97db69cc276fe24ba54f6032c07f0170cc889bd88368d68c65ad97ab755b0889
data

https://www.cybergym.io/assets/data/exploitgym.json

1cf3b94633364a7cb76c35dc502f21fbf889ee02b639f3a20042960a0a9b2533
renderer

https://www.cybergym.io/assets/js/leaderboard.js

7aa2d51c07d2bda8dc8be9910844441aabcfaa03b8b1a9adf1a24e3be45298f2

References

Numerical results are reported by the cited authors.

  1. [1] ExploitGym official leaderboard and benchmark page