NUWA Risk Evaluation · Benchmark
PortBench
NUWA research note · independent review pending
NUWA Research Note
Research questionDesign and assumptionsSource findingsNUWA interpretationLimits and contrary evidenceResearch and decision relevanceInstitutions and rolesOriginal resourcesModel-assisted reading · Selected sections of the original full text or official article; a complete section-by-section audit remains pending.
At a glance
NUWA interpretation: these percentages summarize graded porting performance, not necessarily fully solved task fractions.
Research question
Can agents port command-line tools across languages using a reference implementation?
Design and assumptions
CAISI’s nonpublic benchmark tests cross-language CLI porting with Inspect ReAct and a one-million-weighted-token budget.
Source findings
The report lists GPT-5.5 at 78%, Opus 4.6 at 60% and DeepSeek V4 Pro at 44%.
PortBench uses continuous scoring and was excluded from the report’s binary-success cost comparison.
NUWA interpretation
NUWA interpretation: these percentages summarize graded porting performance, not necessarily fully solved task fractions.
Limits and contrary evidence
The official report promises a fuller benchmark description; task counts, score granularity and full validation remain undisclosed in this source.
Research and decision relevance
Model recommendation: label continuous scoring explicitly and withhold binary success or cost-equivalence interpretations.
Precise source locations
- PortBench · Capability Results: Figure 2Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- PortBench · Benchmarks: PortBenchSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- PortBench · Capability Lag Measurement: token budgetsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
- PortBench · DeepSeek V4 Costs Less Than Other Models Of Similar Capability: exclusionsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
Source checking, extraction, editorial approval and replication are recorded separately. Methodology →
Institutions and roles
Evaluation protocols
Cite and reuse
Cite the original publication for its findings. Cite this NUWA record separately for its explanation and retained snapshot.
Original publication · BibTeX
@misc{portbench,
title = {PortBench},
url = {https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro},
year = {2026},
date = {2026-05-01},
version = {2026-10-25 source capture},
note = {2026-10-25 source capture}
}
NUWA record · BibTeX
@misc{portbench-nuwa,
title = {PortBench — NUWA research record},
url = {https://whitzard.tech/en/nuwa/whitzard-index/work/portbench/},
version = {2026-10-29.1},
organization = {NUWA Frontier AI Safety Lab},
howpublished = {Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--portbench.json},
note = {NUWA research note. Source-reading scope and independent review status are stated on the record. Snapshot 2026-10-29.1. Retained note edition: https://whitzard.tech/nuwa/whitzard-index/data/record-editions/2026-10-29.1--portbench.json.}
}
Evidence snapshot 2026-10-29.1 · Index updates · RSS
References
- [1] PortBench. 2026-05-01 ; Capability Results: Figure 2; Benchmarks: PortBench; Capability Lag Measurement: token budgets