NUWA Evaluation dossier
BountyBench
Evaluates detection, exploitation and patching across real-world codebases, with task outcomes grounded in vulnerability lifecycles.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
BountyBench · Patch success
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
10 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
10 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
10 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 OpenAI Codex CLI: o3-high | 90%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
| OpenAI Codex CLI: o4-mini | 90%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
| Claude Code | 87.5%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
 C-Agent: Claude 3.7 | 60%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
| C-Agent: GPT-4.1 | 50%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
 C-Agent: DeepSeek-R1 | 50%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
 C-Agent: Gemini 2.5 | 45%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
 C-Agent: Llama 4 Maverick | 42.5%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
| C-Agent: o3-high | 35%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
 C-Agent: Qwen3 235B A22B | 25%Patch success | Incomplete | Table 1 · BountyBench— | Not disclosedBountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems | Release —Report — |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Evaluates detection, exploitation and patching across real-world codebases, with task outcomes grounded in vulnerability lifecycles.
- Scoring / outcome
- Detect success · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined.
- Interpretation boundary
- Detection, exploitation and patching have different information conditions and success criteria; bounty amounts do not estimate real-world damage.
A reported evaluation
The source evaluates Claude Code · Table 1 · BountyBench.
Reported measurement: Detect success · 5.0%.
Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. See the source protocol; no independent trial count inferred.
Source-reported measurements for Claude Code in Table 1; values are transcribed, not independently reproduced.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can one framework distinguish detection, exploitation and patching across real applications?
- Task population
- 25 manually configured systems contain forty paid-bounty vulnerabilities; task variants change the information provided and score detect/exploit/patch separately.
- Scoring and failure modes
- Detect, Exploit and Patch pass criteria differ; reported best-of-up-to-three attempts.
- System conditions
- 25 systems and forty bounties with task-specific information affordances.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: With up to three attempts, top reported scores included 12.5% Detect, 67.5% Exploit and 90% Patch from different agent setups.
- Validity and interpretation
- Added-vulnerability coverage and incomplete invariants can miss new defects or incomplete patches; environment preparation is heavily manual.
- Appropriate research use
- Model recommendation: keep task-specific agents, attempts and validation coverage instead of aggregating into one cyber score.
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems · AbstractSelected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems · §4 ExperimentsSelected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems · §6 DiscussionSelected original resource https://arxiv.org/html/2505.15216; sha256:20b337b2f78d1ea058b3ca071004daead3ff8f1bb4b8f84fdbfbbdd74256ab52; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
Actual evaluators: Not disclosed
10 result records · 10 reported system names
Protocols and comparison decisions (3)
BountyBench: Detect success · BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systemssource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Detect success
0–100%Claude Code
Claude Code · 5.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
5.0%OpenAI Codex CLI: o3-highOpenAI Codex CLI: o3-high · 12.5% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
12.5%OpenAI Codex CLI: o4-miniOpenAI Codex CLI: o4-mini · 5.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
5.0%C-Agent: o3-highC-Agent: o3-high · 0.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
0.0%C-Agent: GPT-4.1C-Agent: GPT-4.1 · 0.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
0.0%C-Agent: Gemini 2.5C-Agent: Gemini 2.5 · 2.5% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
2.5%C-Agent: Claude 3.7C-Agent: Claude 3.7 · 5.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
5.0%C-Agent: Qwen3 235B A22BC-Agent: Qwen3 235B A22B · 0.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
0.0%C-Agent: Llama 4 MaverickC-Agent: Llama 4 Maverick · 0.0% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
0.0%C-Agent: DeepSeek-R1C-Agent: DeepSeek-R1 · 2.5% · Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. · See the source protocol; no independent trial count inferred.
2.5%Showing 10 of 10 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 1: detect, exploit and patch success rates, up to three attempts per task. Scaffold differences are retained; metrics are not combined. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- Claude Code
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- OpenAI Codex CLI: o3-high
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- OpenAI Codex CLI: o4-mini
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: o3-high
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: GPT-4.1
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: Gemini 2.5
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: Claude 3.7
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: Qwen3 235B A22B
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: Llama 4 Maverick
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
- C-Agent: DeepSeek-R1
- See the source protocol; no independent trial count inferred. · Table 1 · BountyBench
- modelVersion: not specified · harness: not specified · tools: not specified · budget: not specified
| System / configuration | Detect success | Exploit success | Patch success | Publication / verification |
|---|---|---|---|---|
| Claude CodeHarness not specified | 5.0% | 57.5% | 87.5% | Source checked2026-10-25 |
| OpenAI Codex CLI: o3-highHarness not specified | 12.5% | 47.5% | 90.0% | Source checked2026-10-25 |
| OpenAI Codex CLI: o4-miniHarness not specified | 5.0% | 32.5% | 90.0% | Source checked2026-10-25 |
| C-Agent: o3-highHarness not specified | 0.0% | 37.5% | 35.0% | Source checked2026-10-25 |
| C-Agent: GPT-4.1Harness not specified | 0.0% | 55.0% | 50.0% | Source checked2026-10-25 |
| C-Agent: Gemini 2.5Harness not specified | 2.5% | 40.0% | 45.0% | Source checked2026-10-25 |
| C-Agent: Claude 3.7Harness not specified | 5.0% | 67.5% | 60.0% | Source checked2026-10-25 |
| C-Agent: Qwen3 235B A22BHarness not specified | 0.0% | 17.5% | 25.0% | Source checked2026-10-25 |
| C-Agent: Llama 4 MaverickHarness not specified | 0.0% | 42.5% | 42.5% | Source checked2026-10-25 |
| C-Agent: DeepSeek-R1Harness not specified | 2.5% | 37.5% | 50.0% | Source checked2026-10-25 |
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Andy K. Zhang, Joey Ji, et al.. BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems. 2025-05-21. v3 [arXiv]; Author affiliations on the original paper title page; Table 1