{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"aisi-cyber-scenarios","title":"Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios","aliases":[],"authors":["Linus Folkerts","Will Payne","Simon Inman","Philippos Giavridis","Joe Skinner","Sam Deverett","James Aung","Ekin Zorer","Michael Schmatz","Mahmoud Ghanem","John Wilkinson","Alan Steer","Vy Hong","Jessica Wang"],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"study","sourceId":"aisi-cyber-range","attributionSourceId":"aisi-cyber-range-author-source","originalUrl":"https://www.aisi.gov.uk/research/measuring-ai-agents-progress-on-multi-step-cyber-attack-scenarios","publicationDate":"2026-03-16","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Primary dimension follows task completion, with budget and scenario constraints.","zh":"主维度依据任务完成情况，须同时呈现预算及场景约束。"},"protocolIds":["aisi-cyber-scenarios"],"contributions":[{"organizationId":"aisi","role":"publisher","sourceId":"aisi-cyber-range","locator":"Official publication masthead and article byline"}],"summary":{"en":"Assesses autonomous progress through multi-step cyber scenarios.","zh":"评估智能体在多步网络场景中的自主推进。"},"note":{"question":{"en":"Task progress in two defined ranges under varied inference budgets.","zh":"在两套明确靶场及不同推理预算下的任务推进。"},"method":{"en":"Agents in two purpose-built cyber ranges","zh":"两套专设网络靶场中的智能体"},"finding":{"en":"Assesses autonomous progress through multi-step cyber scenarios.","zh":"评估智能体在多步网络场景中的自主推进。"},"supports":{"en":"Primary dimension follows task completion, with budget and scenario constraints.","zh":"主维度依据任务完成情况，须同时呈现预算及场景约束。"},"limits":{"en":"A purpose-built study is not an open-ended estimate of real-world cyber attack success.","zh":"专设研究不代表现实网络攻击成功概率。"},"connections":null},"resources":[{"label":"Read the full paper","url":"https://arxiv.org/abs/2603.11214"}],"noteStatus":{"question":"limited","method":"limited","finding":"limited","supports":"limited","limits":"limited","connections":"limited"},"noteAvailability":{"en":"Existing source-specific draft sections are retained with limited review status. A complete methods, findings and limitations check remains pending; generic sections were omitted.","zh":"已有的来源相关草稿章节予以保留，审核状态为有限核查。完整的方法、发现与限制核查仍待完成；通用套话章节已省略。"},"noteSourceLocator":"Retained existing source-specific note sections; this pass audits duplicate/boilerplate structure and source availability, not complete section-level verification.","tags":[]},"reading":{"id":"aisi-cyber-scenarios","title":"Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios","en":{"question":"How do model generations and inference budgets affect progress on linked cyber-range tasks?","design":"Seven models are tested on two purpose-built ranges with 32 and 7 steps; the recent-model 100M-token end-to-end conditions have five attempts each.","findings":["At 10M tokens, mean corporate-range progress rose from 1.7 steps for GPT-4o to 9.8 for Opus 4.6.","The best reported run completed 22/32 steps; this is not the average completion rate."],"interpretation":"NUWA interpretation: budget and model generation both matter for cumulative task progress.","limits":"Two bespoke ranges and small high-budget samples limit generalization; publication can reduce future held-out validity (§5).","decisionRelevance":"Model recommendation: compare matched token budgets and report run variance alongside best-run progress.","takeaway":"NUWA interpretation: budget and model generation both matter for cumulative task progress.","connections":[],"scoring":"Number of range steps completed; best-run and mean progress differ.","conditions":"10M versus 100M-token conditions; recent high-budget end-to-end cells have five attempts.","statistics":"At 10M tokens, mean corporate-range progress rose from 1.7 steps for GPT-4o to 9.8 for Opus 4.6. The best reported run completed 22/32 steps; this is not the average completion rate.","usage":"Model recommendation: compare matched token budgets and report run variance alongside best-run progress."},"zh":{"question":"模型代际与推理预算如何影响连续靶场任务进展？","design":"七个模型在32步、7步两个定制靶场测试；近期模型的100M令牌端到端条件各运行五次。","findings":["10M令牌下，企业靶场平均进展从GPT-4o的1.7步升至Opus 4.6的9.8步。","最佳单次完成22/32步，并非平均完成率。"],"interpretation":"NUWA 解读：预算与模型代际均影响累积任务进展。","limits":"仅两个定制靶场，高预算样本较小；披露还可能削弱未来保留集有效性（§5）。","decisionRelevance":"模型建议：匹配令牌预算，并在最佳进展旁报告运行差异。","takeaway":"NUWA 解读：预算与模型代际均影响累积任务进展。","connections":[],"scoring":"按完成靶场步骤数计，最佳运行不同于平均进展。","conditions":"区分10M与100M令牌，近期高预算端到端单元五次尝试。","statistics":"10M令牌下，企业靶场平均进展从GPT-4o的1.7步升至Opus 4.6的9.8步。 最佳单次完成22/32步，并非平均完成率。","usage":"模型建议：匹配令牌预算，并在最佳进展旁报告运行差异。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"aisi-cyber-range-author-source","url":"https://arxiv.org/html/2603.11214","locator":"Abstract","version":"Selected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established","artifactSha256":"fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf"},{"sourceId":"aisi-cyber-range-author-source","url":"https://arxiv.org/html/2603.11214","locator":"§3.3 Experimental design","version":"Selected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established","artifactSha256":"fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf"},{"sourceId":"aisi-cyber-range-author-source","url":"https://arxiv.org/html/2603.11214","locator":"§2 Results","version":"Selected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established","artifactSha256":"fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf"},{"sourceId":"aisi-cyber-range-author-source","url":"https://arxiv.org/html/2603.11214","locator":"§5 Limitations","version":"Selected original resource https://arxiv.org/html/2603.11214; sha256:fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf; archived source capture; exact capture time not independently established","artifactSha256":"fe735767bb4d07e484ce73b74b4095cb5dda946262a39b0fd70c3ed2f05015cf"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"fb20a4589a28c51458efaa32f6ac1994ff2e11e03ba9f8bdab9e084772e7324a"}},"sources":[{"id":"aisi-cyber-range","title":"Measuring AI Agents’ Progress on Multi-Step Cyber Attack Scenarios","url":"https://www.aisi.gov.uk/research/measuring-ai-agents-progress-on-multi-step-cyber-attack-scenarios","version":"Research page checked 2026-10-25","publicationDate":null,"datePrecision":"not recorded in catalogue","type":"research-report","verifiedOn":"2026-10-25","documentId":"https://www.aisi.gov.uk/research/measuring-ai-agents-progress-on-multi-step-cyber-attack-scenarios","authors":["Linus Folkerts","Will Payne","Simon Inman","Philippos Giavridis","Joe Skinner","Sam Deverett †","James Aung","Ekin Zorer","Michael Schmatz","Mahmoud Ghanem","John Wilkinson","Alan Steer","Vy Hong","Jessica Wang"]}]}