Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

How do model generations and inference budgets affect progress on linked cyber-range tasks?

Task population

Seven models are tested on two purpose-built ranges with 32 and 7 steps; the recent-model 100M-token end-to-end conditions have five attempts each.

Scoring and failure modes

Number of range steps completed; best-run and mean progress differ.

System conditions

10M versus 100M-token conditions; recent high-budget end-to-end cells have five attempts.

Statistical meaning

Source-reported result statements; this does not establish trial counts, denominators or an estimator: At 10M tokens, mean corporate-range progress rose from 1.7 steps for GPT-4o to 9.8 for Opus 4.6. The best reported run completed 22/32 steps; this is not the average completion rate.

Validity and interpretation

Two bespoke ranges and small high-budget samples limit generalization; publication can reduce future held-out validity (§5).

Appropriate research use

Model recommendation: compare matched token budgets and report run variance alongside best-run progress.

• Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios · AbstractSelected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established
• Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios · §3.3 Experimental designSelected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established
• Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios · §2 ResultsSelected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established
• Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios · §5 LimitationsSelected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established