{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"ctf-archive-diamond","title":"CTF-Archive-Diamond","aliases":["CTF-Archive-Diamond"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"ctf-archive-diamond-primary","attributionSourceId":"ctf-archive-diamond-primary","originalUrl":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","publicationDate":"2026-05-01","updatedOn":"2026-10-25","areas":["cybersecurity"],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["ctf-archive-diamond"],"contributions":[],"summary":{"en":"How do agents solve a difficult CTF subset?","zh":"智能体在高难度 CTF 子集中表现如何？"},"note":{"question":{"en":"How do agents solve a difficult CTF subset?","zh":"智能体在高难度 CTF 子集中表现如何？"},"method":{"en":"CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately.","zh":"CAISI 采用 pwn.challenge 集合的 285 项高难度 CTF 挑战；模型插补结果单独标记。"},"finding":null,"supports":null,"limits":{"en":"CTF tasks are not live operational attacks; subset-imputed values are not directly observed full-set success rates.","zh":"CTF 不是现实行动攻击；子集插补值不是全量直接观察的成功率。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"ctf-archive-diamond","title":"CTF-Archive-Diamond","en":{"question":"How do models perform on CAISI’s harder CTF selection under a controlled agent budget?","design":"The official report describes 285 pwn.challenge-derived tasks, Inspect ReAct and a one-million-weighted-token budget.","findings":["The report lists GPT-5.5 at 71%, Opus 4.6 at 46% and DeepSeek V4 Pro at 32%.","DeepSeek’s marked value is IRT-imputed from a subset, not directly observed on every task."],"interpretation":"NUWA interpretation: the 285-task benchmark size is not the direct DeepSeek trial denominator.","limits":"The report gives a short benchmark description; held-out provenance, all run counts and raw imputation inputs are not audited here.","decisionRelevance":"Model recommendation: preserve the imputation footnote and budget before comparing with independently run CTF suites.","takeaway":"NUWA interpretation: the 285-task benchmark size is not the direct DeepSeek trial denominator.","connections":[],"scoring":"Task solve rate; the marked DeepSeek V4 score is IRT-imputed from a subset.","conditions":"285 CTF tasks; Inspect ReAct, 1M weighted tokens.","statistics":"The report lists GPT-5.5 at 71%, Opus 4.6 at 46% and DeepSeek V4 Pro at 32%.","usage":"Model recommendation: preserve the imputation footnote and budget before comparing with independently run CTF suites."},"zh":{"question":"受控预算下模型在CAISI较难CTF集合表现如何？","design":"官网报告描述285道pwn.challenge来源题，使用Inspect ReAct及百万加权令牌预算。","findings":["报告列GPT-5.5为71%，Opus 4.6为46%，DeepSeek V4 Pro为32%。","DeepSeek标星结果由子样本IRT推补，不是全题直接观测。"],"interpretation":"NUWA 解读：285题规模不是DeepSeek直接试验分母。","limits":"报告仅简述基准，此处未审查全部试次、保留来源及推补输入。","decisionRelevance":"模型建议：比较独立CTF结果前保留推补注释与预算。","takeaway":"NUWA 解读：285题规模不是DeepSeek直接试验分母。","connections":[],"scoring":"解题率，标星DeepSeek V4由子样本IRT推补。","conditions":"285题，Inspect ReAct及百万加权令牌。","statistics":"报告列GPT-5.5为71%，Opus 4.6为46%，DeepSeek V4 Pro为32%。","usage":"模型建议：比较独立CTF结果前保留推补注释与预算。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"ctf-archive-diamond-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Capability Results: Figure 2 and *** footnote","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"},{"sourceId":"ctf-archive-diamond-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Benchmarks: CTF-Archive-Diamond","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"},{"sourceId":"ctf-archive-diamond-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Capability Lag Measurement: token budgets","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"},{"sourceId":"ctf-archive-diamond-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Appendix: IRT","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"97e476571786fd83407fee0cd7bb0e0fa5f05e3933fa772f8de03d237fa8181d"}},"sources":[{"id":"ctf-archive-diamond-primary","title":"CTF-Archive-Diamond","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","version":"2026-10-25 source capture","publicationDate":"2026-05-01","datePrecision":"day","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro"}]}