Evaluation dossier

PortBench

Can models port CLI tools across programming languages?

Affiliation not specified / 2026-10-25 source capture / Capability

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

PortBench · Reported task accuracy

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

4 observations · 4 dated

● Matched◇ Configuration differs□ Some conditions undisclosed

Select a point or table row to inspect its conditions and provenance.

4 observations

Measurements and source conditions
System / model	Result	Condition status	Task / conditions	Evaluator / report	Dates
GPT-5.5 (xhigh)GPT-5.5	78%Reported task accuracy	Incomplete	PortBench1M weighted tokensInspect ReAct	NIST / CAISICAISI Evaluation of DeepSeek V4 Pro	Release 2026-04-23Report 2026-05-01
Claude Opus 4.6 (max)Claude Opus 4.6	60%Reported task accuracy	Incomplete	PortBench1M weighted tokensInspect ReAct	NIST / CAISICAISI Evaluation of DeepSeek V4 Pro	Release 2026-02-05Report 2026-05-01
DeepSeek V4 Pro (max)DeepSeek V4 Pro	44%Reported task accuracy	Incomplete	PortBench1M weighted tokensInspect ReAct	NIST / CAISICAISI Evaluation of DeepSeek V4 Pro	Release 2026-04-24Report 2026-05-01
GPT-5.4 mini (xhigh)GPT-5.4 mini	41%Reported task accuracy	Incomplete	PortBench1M weighted tokensInspect ReAct	NIST / CAISICAISI Evaluation of DeepSeek V4 Pro	Release 2026-03-17Report 2026-05-01

Task and measurement

Benchmark protocol profile

Task / subject: The model or agent in the stated task configuration.
Measurement target: CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.
Scoring / outcome: Reported task accuracy · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.
Interpretation boundary: The task set is non-public and continuous scoring differs from binary task success.

CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.

A reported evaluation

The source evaluates GPT-5.5 (xhigh) · PortBench.

Run configuration: Inspect ReAct · 1M weighted tokens.

Reported measurement: Reported task accuracy · 78%.

CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. Non-public CAISI task set; report describes continuous scoring.

Source-reported measurements in Capability Results; Figure 2. Transcription is not independent experimental replication.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct: Can agents port command-line tools across languages using a reference implementation?
Task population: CAISI’s nonpublic benchmark tests cross-language CLI porting with Inspect ReAct and a one-million-weighted-token budget.
Scoring and failure modes: Continuous porting score, not an established binary full-task success fraction.
System conditions: Nonpublic CLI language-porting tasks, Inspect ReAct, 1M weighted tokens.
Statistical meaning: Source-reported result statements; this does not establish trial counts, denominators or an estimator: The report lists GPT-5.5 at 78%, Opus 4.6 at 60% and DeepSeek V4 Pro at 44%.
Validity and interpretation: The official report promises a fuller benchmark description; task counts, score granularity and full validation remain undisclosed in this source.
Appropriate research use: Model recommendation: label continuous scoring explicitly and withhold binary success or cost-equivalence interpretations.

PortBench · Capability Results: Figure 2Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
PortBench · Benchmarks: PortBenchSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
PortBench · Capability Lag Measurement: token budgetsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established
PortBench · DeepSeek V4 Costs Less Than Other Models Of Similar Capability: exclusionsSelected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

NIST / CAISI

CAISI Evaluation of DeepSeek V4 Pro →

Actual evaluators: NIST / CAISI

4 result records · 4 reported system names

Protocols and comparison decisions (1)

PortBench: Reported task accuracy · NIST / CAISIsource-report · Scope reviewed

Original source charts and materials

PortBench

Original evaluator results · source order

Reported task accuracy

0–100%

GPT-5.5 (xhigh)

GPT-5.5 (xhigh) · 78% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

78%

GPT-5.4 mini (xhigh)

GPT-5.4 mini (xhigh) · 41% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

41%

Claude Opus 4.6 (max)

Claude Opus 4.6 (max) · 60% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

60%

DeepSeek V4 Pro (max)

DeepSeek V4 Pro (max) · 44% · CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget. · Inspect ReAct · Non-public CAISI task set; report describes continuous scoring.

44%

Showing 4 of 4 systems

CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.

The task set is non-public and continuous scoring differs from binary task success. Report dates and original executor are shown with this assessment.

Evaluation conditions and system configurations

GPT-5.5 (xhigh): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: GPT-5.5 (xhigh) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

GPT-5.4 mini (xhigh): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: GPT-5.4 mini (xhigh) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

Claude Opus 4.6 (max): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: Claude Opus 4.6 (max) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

DeepSeek V4 Pro (max): Non-public CAISI task set; report describes continuous scoring. · PortBench; modelVersion: DeepSeek V4 Pro (max) · harness: Inspect ReAct · tools: Software engineering environment · budget: 1M weighted tokens

PortBench · source order
System / configuration	Reported task accuracy	Publication / verification
GPT-5.5 (xhigh)Inspect ReAct	78%	Published2026-05-01
GPT-5.4 mini (xhigh)Inspect ReAct	41%	Published2026-05-01
Claude Opus 4.6 (max)Inspect ReAct	60%	Published2026-05-01
DeepSeek V4 Pro (max)Inspect ReAct	44%	Published2026-05-01

Assessment provenance

CAISI Evaluation of DeepSeek V4 Pro → · External report

References

[1] deepseek-v4-pro release ; News article title, dated heading and availability
[2] deepseek-v4-pro developer model card ; Introduction; model comparison table
[3] OpenAI — introducing gpt 5 5 ; Publication date and availability section; Parameter counts not disclosed in launch article
[4] OpenAI — introducing gpt 5 4 mini and nano ; Publication date and availability section; Parameter counts not disclosed in launch article
[5] Claude Opus 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
[6] CAISI Evaluation of DeepSeek V4 Pro. 2026-05-01. 1 May 2026 ; Capability Results; Figure 2

Search NUWA Index

PortBench

PortBench · Reported task accuracy

Task and measurement

A reported evaluation

Evidence across reports

CAISI Evaluation of DeepSeek V4 Pro →

PortBench

Reported task accuracy

Assessment provenance

Related research questions

Autonomy / Sustained autonomous work →

References