DeepSeek R1

Method and scope

Measures agents’ exploitation of real web-application vulnerabilities in runnable environments with explicit success checks.

Evidence type

Benchmark

Assessment subject

Model or agent system under the original task configuration

Primary dimension

Capability

Evidence snapshot

2026-10-29.1

The main dimension follows the task outcome in the original protocol. Topic relevance is documented in the parent research record; no unreviewed question-level links are inferred.

Source finding

Reported benchmark performance: 26.7 percent. Conditions are retained from the source.

Limitations

Disclosed vulnerabilities and bounded environments differ from open-ended discovery and enterprise intrusion.

Report labels these as accuracy / percentage of tasks solved. Appendix A3 describes mixed-effects logistic regression estimates and model-based standard errors; do not reconstruct empirical successes or treat this as raw pass@1 counts. No exact API snapshot identifier or per-model trial count disclosed here; benchmark task count is not number of repeated trials. Partial benchmark/harness-specific performance, not measured autonomous real-world risk. Whole-task sandbox benchmark success; not a measurement of all cyber capability or actual harm. CAISI modified these benchmarks and agent setup.

Original figure retained without estimating numerical values. Full source tables have not been transcribed.

Related research questions