NUWA Evaluation dossier
EVMbench
Can agents detect, repair and exploit smart-contract vulnerabilities in the benchmark?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
EVMbench · Detect
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
1 observations · 1 dated
Select a point or table row to inspect its conditions and provenance.
1 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 DeepSeek v4 ProDeepSeek V4 Pro | 24.4%Detect | Incomplete | EVMbenchdirect API, xhigh— | Neo ResearchEvaluating DeepSeek v4 Pro for Frontier Risks | Release 2026-04-24Report 2026-05-29 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- The model or agent in the stated task configuration.
- Measurement target
- Separates detection, patching and exploitation modes with their stated task sets.
- Scoring / outcome
- Detect · Source-reported measurement under the recorded configuration.
- Interpretation boundary
- Modes and task subsets cannot be combined into an undifferentiated cyber-risk score.
Separates detection, patching and exploitation modes with their stated task sets.
A reported evaluation
The source evaluates DeepSeek v4 Pro · EVMbench.
Reported measurement: Detect · 24.4%.
Source-reported measurement under the recorded configuration. Per-result sample size not disclosed
Detect: 24.4 percent. Conditions are retained from the source.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can agents detect, patch and exploit historical smart-contract vulnerabilities?
- Task population
- 117 curated vulnerabilities from forty audits support separate modes; isolated-chain execution and deterministic transaction replay validate exploit tasks.
- Scoring and failure modes
- Separate detection recall, patch validation and programmatically verified exploit score.
- System conditions
- 117 vulnerabilities from forty audits; isolated local single-chain replay.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: GPT-5.3-Codex via Codex CLI scored 71.0% in exploit mode, versus GPT-5’s 33.3%.
- Validity and interpretation
- Known-ground-truth detection misses novel issues; sequential replay excludes timing-dependent behavior, and local single-chain tests limit realism.
- Appropriate research use
- Model recommendation: cite mode and harness conditions; do not treat exploit scores as live-network compromise rates.
- EVMbench · Introducing EVMbench: capability modes (official article paragraphs 31–44)Selected original resource https://openai.com/index/introducing-evmbench/; sha256:0d7b334d1452cbe75c15509ed172d187c3c9e70ce186f39678761a9e9ac8ddaa; web-tool-extracted-response; not original HTML
- EVMbench · Limitations (paragraphs 45–49)Selected original resource https://openai.com/index/introducing-evmbench/; sha256:0d7b334d1452cbe75c15509ed172d187c3c9e70ce186f39678761a9e9ac8ddaa; web-tool-extracted-response; not original HTML
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Neo Research
Evaluating DeepSeek v4 Pro for Frontier Risks →
Actual evaluators: Neo Research
4 result records · 1 reported system names
Protocols and comparison decisions (4)
Original source charts and materials
Protocol scope is catalogued. Exact numerical extraction and an appropriate source figure remain to be reviewed.
Assessment provenance
Evaluating DeepSeek v4 Pro for Frontier Risks → · External report
Related research questions
Cybersecurity / Cyber task execution →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
- [1] deepseek-v4-pro release ; News article title, dated heading and availability
- [2] deepseek-v4-pro developer model card ; Introduction; model comparison table
- [3] Evaluating DeepSeek v4 Pro for Frontier Risks. 2026-05-29. v1.1 · 29 May 2026 ; PDF p. 9; EVMbench; Detect; PDF p. 9; EVMbench; Patch; PDF p. 9; EVMbench; Exploit
- [4] EVMbench