Risk Evaluation · Benchmark

CTF-Archive-Diamond

NUWA research note · independent review pending

First published: 2026-05-01Source checked: 2026-10-25

Original source ↗Cite this record ↓Cybersecurity →

NUWA Research Note

Research question Design and assumptions Source findings NUWA interpretation Limits and contrary evidence Research and decision relevance Institutions and roles Original resources

Model-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.

At a glance

NUWA interpretation: the 285-task benchmark size is not the direct DeepSeek trial denominator.

Research question

How do models perform on CAISI’s harder CTF selection under a controlled agent budget?

Design and assumptions

The official report describes 285 pwn.challenge-derived tasks, Inspect ReAct and a one-million-weighted-token budget.

Source findings

The report lists GPT-5.5 at 71%, Opus 4.6 at 46% and DeepSeek V4 Pro at 32%.

DeepSeek’s marked value is IRT-imputed from a subset, not directly observed on every task.

NUWA interpretation

NUWA interpretation: the 285-task benchmark size is not the direct DeepSeek trial denominator.

Limits and contrary evidence

The report gives a short benchmark description; held-out provenance, all run counts and raw imputation inputs are not audited here.

Research and decision relevance

Model recommendation: preserve the imputation footnote and budget before comparing with independently run CTF suites.

Precise source locations

CTF-Archive-Diamond · Capability Results: Figure 2 and *** footnoteSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
CTF-Archive-Diamond · Benchmarks: CTF-Archive-DiamondSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
CTF-Archive-Diamond · Capability Lag Measurement: token budgetsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
CTF-Archive-Diamond · Appendix: IRTSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established

Source checking, extraction, editorial approval and replication are recorded separately. Methodology →

Institutions and roles

Affiliation not specified

Evaluation protocols

CTF-Archive-Diamond →

Cite and reuse

Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.

BibTeX ↓RIS ↓Record + source metadata · JSON ↓Reading coverage · JSON Suggest a correction

Original publication · BibTeX

@misc{ctf-archive-diamond,
  title = {CTF-Archive-Diamond},
  url = {https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro},
  year = {2026},
  date = {2026-05-01},
  version = {2026-10-25 source capture},
  note = {2026-10-25 source capture}
}

NUWA record · BibTeX

@misc{ctf-archive-diamond-nuwa,
  title = {CTF-Archive-Diamond — NUWA research record},
  url = {https://whitzard.tech/en/nuwa/whitzard-index/work/ctf-archive-diamond/},
  version = {2026-10-29.1},
  organization = {NUWA Frontier AI Safety Lab},
  howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--ctf-archive-diamond.json},
  note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--ctf-archive-diamond.json.}
}

Evidence snapshot 2026-10-29.1 · Index updates · RSS

References

[1] CTF-Archive-Diamond. 2026-05-01 ; Capability Results: Figure 2 and *** footnote; Benchmarks: CTF-Archive-Diamond; Capability Lag Measurement: token budgets