NUWA Evaluation dossier
CTF-Archive-Diamond
How do agents solve a difficult CTF subset?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
CTF-Archive-Diamond · benchmark_score
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
7 observations · 7 dated
Select a point or table row to inspect its conditions and provenance.
7 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 OpenAI GPT-5.5GPT-5.5 | 71%benchmark_score | Incomplete | CTF-Archive-Diamondxhigh1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-04-23Report 2026-05-01 |
| GPT-5.4 | 54.4%benchmark_score | Incomplete | CTF-Archive Diamondxhigh1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-03-05Report 2026-05-01 |
 Anthropic Opus 4.6Claude Opus 4.6 | 46%benchmark_score | Incomplete | CTF-Archive-Diamondmax1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-02-05Report 2026-05-01 |
| Claude Opus 4.6 | 45.6%benchmark_score | Incomplete | CTF-Archive Diamondmax1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-02-05Report 2026-05-01 |
| OpenAI GPT-5.4 miniGPT-5.4 mini | 32%benchmark_score | Incomplete | CTF-Archive-Diamondxhigh1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-03-17Report 2026-05-01 |
 DeepSeek V4 Pro | 32%benchmark_score | Incomplete | CTF-Archive-Diamondmax1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-04-24Report 2026-05-01 |
| DeepSeek V4 Pro | 31.7%benchmark_score | Incomplete | CTF-Archive Diamondmax1M weighted tokensInspect built-in ReAct agent | NIST / CAISICAISI Evaluation of DeepSeek V4 Pro | Release 2026-04-24Report 2026-05-01 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately.
- Scoring / outcome
- Task accuracy · CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately.
- Interpretation boundary
- CTF tasks are not live operational attacks; subset-imputed values are not directly observed full-set success rates.
CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately.
A reported evaluation
The source evaluates GPT-5.5 (xhigh) · CTF-Archive-Diamond.
Run configuration: Inspect ReAct · 1M weighted tokens.
Reported measurement: Task accuracy · 71%.
CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately. 285 difficult CTF tasks; DeepSeek score is imputed from a subset.
Source-reported measurements in Capability Results; Figure 2; *** footnote. Transcription is not independent experimental replication.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- How do models perform on CAISI’s harder CTF selection under a controlled agent budget?
- Task population
- The official report describes 285 pwn.challenge-derived tasks, Inspect ReAct and a one-million-weighted-token budget.
- Scoring and failure modes
- Task solve rate; the marked DeepSeek V4 score is IRT-imputed from a subset.
- System conditions
- 285 CTF tasks; Inspect ReAct, 1M weighted tokens.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: The report lists GPT-5.5 at 71%, Opus 4.6 at 46% and DeepSeek V4 Pro at 32%.
- Validity and interpretation
- The report gives a short benchmark description; held-out provenance, all run counts and raw imputation inputs are not audited here.
- Appropriate research use
- Model recommendation: preserve the imputation footnote and budget before comparing with independently run CTF suites.
- CTF-Archive-Diamond · Capability Results: Figure 2 and *** footnoteSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CTF-Archive-Diamond · Benchmarks: CTF-Archive-DiamondSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CTF-Archive-Diamond · Capability Lag Measurement: token budgetsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- CTF-Archive-Diamond · Appendix: IRTSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
NIST / CAISI
CAISI Evaluation of DeepSeek V4 Pro →
Actual evaluators: NIST / CAISI
11 result records · 10 reported system names
Protocols and comparison decisions (6)
Original source charts and materials
CTF-Archive-Diamond
Original evaluator results · source order
Select systems
Task accuracy
0–100%GPT-5.5 (xhigh)GPT-5.4 mini (xhigh)Claude Opus 4.6 (max)DeepSeek V4 Pro (max)CAISI uses 285 difficult CTF challenges from the pwn.challenge collection; imputed model results are marked separately.
CTF tasks are not live operational attacks; subset-imputed values are not directly observed full-set success rates. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- GPT-5.5 (xhigh)
- 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond
- modelVersion: GPT-5.5 (xhigh) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens
- GPT-5.4 mini (xhigh)
- 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond
- modelVersion: GPT-5.4 mini (xhigh) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens
- Claude Opus 4.6 (max)
- 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond
- modelVersion: Claude Opus 4.6 (max) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens
- DeepSeek V4 Pro (max)
- 285 difficult CTF tasks; DeepSeek score is imputed from a subset. · CTF-Archive-Diamond
- modelVersion: DeepSeek V4 Pro (max) · harness: Inspect ReAct · tools: CTF task environment · budget: 1M weighted tokens
| System / configuration | Task accuracy | Publication / verification |
|---|---|---|
| GPT-5.5 (xhigh)Inspect ReAct | 71% | Published2026-05-01 |
| GPT-5.4 mini (xhigh)Inspect ReAct | 32% | Published2026-05-01 |
| Claude Opus 4.6 (max)Inspect ReAct | 46% | Published2026-05-01 |
| DeepSeek V4 Pro (max)Inspect ReAct | ≈32%IRT-imputed, not directly observed. | Published2026-05-01 |
Assessment provenance
CAISI Evaluation of DeepSeek V4 Pro → · External report
Related research questions
Cybersecurity / Cyber task execution →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] deepseek-v4-pro release ; News article title, dated heading and availability
- [2] deepseek-v4-pro developer model card ; Introduction; model comparison table
- [3] OpenAI — introducing gpt 5 5 ; Publication date and availability section; Parameter counts not disclosed in launch article
- [4] OpenAI — introducing gpt 5 4 ; Publication date and availability section; Parameter counts not disclosed in launch article
- [5] OpenAI — introducing gpt 5 4 mini and nano ; Publication date and availability section; Parameter counts not disclosed in launch article
- [6] Claude Opus 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
- [7] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026 ; Capability Results; Figure 2; *** footnote; Figure 2 HTML table, CTF-Archive-Diamond, OpenAI GPT-5.5 (xhigh); Figure 2 HTML table, CTF-Archive-Diamond, OpenAI GPT-5.4 mini (xhigh)