NUWA Evidence record
DeepSeek V4 Pro (max)
PortBench · Evidence record
Original source charts and materials
PortBench
Original evaluator results · source order
Select systems
Reported task accuracy
0–100%
DeepSeek V4 Pro (max)CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.
The task set is non-public and continuous scoring differs from binary task success. Report dates and original executor are shown with this assessment.
Evaluation conditions and system configurations
- DeepSeek V4 Pro (max)
- Non-public CAISI task set; report describes continuous scoring. · PortBench
- modelVersion: DeepSeek V4 Pro (max) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens
| System / configuration | Reported task accuracy | Publication / verification |
|---|---|---|
| DeepSeek V4 Pro (max)Inspect ReAct | 44% | Published2026-05-01 |
Assessment provenance
CAISI Evaluation of DeepSeek V4 Pro → · External report
Actual evaluation: NIST / CAISI · Model record →
Method and scope
CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.
- Evidence type
- Benchmark
- Assessment subject
- The model or agent in the stated task configuration.
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
The main dimension follows the stated measurement; contextual risk relevance does not establish harmful deployment capability.
Source finding
Source-reported measurements in Capability Results; Figure 2. Transcription is not independent experimental replication.
Limitations
The task set is non-public and continuous scoring differs from binary task success.
The task set is non-public and continuous scoring differs from binary task success.
Related research questions
Autonomy / Sustained autonomous work →
The protocol explicitly measures this question or an enabling sub-capability; conclusions retain its stated conditions.
References
Numerical results are reported by the cited authors.
- [1] PortBench. 2026-05-01. 1 May 2026 ; Capability Results; Figure 2