{"schemaVersion":"nuwa-current-record-1","version":"2026-10-29.1","publication":"published","work":{"id":"portbench","title":"PortBench","aliases":["PortBench"],"authors":[],"primaryFunction":"evaluation","secondaryFunctions":[],"kind":"benchmark","sourceId":"portbench-primary","attributionSourceId":"portbench-primary","originalUrl":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","publicationDate":"2026-05-01","updatedOn":"2026-10-25","areas":["autonomy"],"scopeNote":{"en":"Mapped to the documented research question. A risk-topic association is not a severity score.","zh":"依据明确研究问题关联；风险主题不代表严重性评分。"},"protocolIds":["portbench"],"contributions":[],"summary":{"en":"Can models port CLI tools across programming languages?","zh":"模型能否跨编程语言移植 CLI 工具？"},"note":{"question":{"en":"Can models port CLI tools across programming languages?","zh":"模型能否跨编程语言移植 CLI 工具？"},"method":{"en":"CAISI’s held-out software engineering evaluation uses reference implementations and a reported agent budget.","zh":"CAISI 的保留软件工程评测采用参考实现和明确智能体预算。"},"finding":null,"supports":null,"limits":{"en":"The task set is non-public and continuous scoring differs from binary task success.","zh":"任务集未公开，连续评分不同于二元任务成功。"},"connections":null},"noteAvailability":{"en":"This record describes the protocol. Report-specific outcomes are linked through assessments; no unsourced general finding is added.","zh":"本档案说明协议。报告特定结果通过评估记录关联，不补写无出处的总体发现。"},"resources":[],"tags":[]},"reading":{"id":"portbench","title":"PortBench","en":{"question":"Can agents port command-line tools across languages using a reference implementation?","design":"CAISI’s nonpublic benchmark tests cross-language CLI porting with Inspect ReAct and a one-million-weighted-token budget.","findings":["The report lists GPT-5.5 at 78%, Opus 4.6 at 60% and DeepSeek V4 Pro at 44%.","PortBench uses continuous scoring and was excluded from the report’s binary-success cost comparison."],"interpretation":"NUWA interpretation: these percentages summarize graded porting performance, not necessarily fully solved task fractions.","limits":"The official report promises a fuller benchmark description; task counts, score granularity and full validation remain undisclosed in this source.","decisionRelevance":"Model recommendation: label continuous scoring explicitly and withhold binary success or cost-equivalence interpretations.","takeaway":"NUWA interpretation: these percentages summarize graded porting performance, not necessarily fully solved task fractions.","connections":[],"scoring":"Continuous porting score, not an established binary full-task success fraction.","conditions":"Nonpublic CLI language-porting tasks, Inspect ReAct, 1M weighted tokens.","statistics":"The report lists GPT-5.5 at 78%, Opus 4.6 at 60% and DeepSeek V4 Pro at 44%.","usage":"Model recommendation: label continuous scoring explicitly and withhold binary success or cost-equivalence interpretations."},"zh":{"question":"智能体能否依据参考实现跨语言移植命令行工具？","design":"CAISI私有基准测CLI跨语言移植，使用Inspect ReAct与百万加权令牌预算。","findings":["报告列GPT-5.5为78%，Opus 4.6为60%，DeepSeek V4 Pro为44%。","PortBench连续计分，未纳入二元成功成本比较。"],"interpretation":"NUWA 解读：百分比汇总分级移植表现，不一定是完整解题比例。","limits":"官网表示将另行详述，现来源未披露题数、评分粒度与完整验证。","decisionRelevance":"模型建议：明确标注连续分，不作二元成功或成本等价解释。","takeaway":"NUWA 解读：百分比汇总分级移植表现，不一定是完整解题比例。","connections":[],"scoring":"连续移植分，不是已确立的二元全解比例。","conditions":"私有CLI语言移植任务，Inspect ReAct及百万加权令牌。","statistics":"报告列GPT-5.5为78%，Opus 4.6为60%，DeepSeek V4 Pro为44%。","usage":"模型建议：明确标注连续分，不作二元成功或成本等价解释。"},"assertionPolicy":{"question":"source-research-question","design":"source-finding","findings":"source-finding","interpretation":"nuwa-interpretation","limits":"source-limit-or-explicit-editorial-scope-limit","connections":"source-supported-relationship-only","decisionRelevance":"model-recommendation-not-source-conclusion"},"sourceRefs":[{"sourceId":"portbench-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Capability Results: Figure 2","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"},{"sourceId":"portbench-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Benchmarks: PortBench","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"},{"sourceId":"portbench-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"Capability Lag Measurement: token budgets","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"},{"sourceId":"portbench-primary","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","locator":"DeepSeek V4 Costs Less Than Other Models Of Similar Capability: exclusions","version":"Selected original resource https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro; sha256:37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623; archived source capture; exact capture time not independently established","artifactSha256":"37f75aab5640e04d198b94a0ca57704b06d7823caeeb05c4280c8fc461b00623"}],"publication":{"state":"published","publishedOn":"2026-10-29","authorizationReference":"user-authorized-global-index-publication-2026-10-29","reviewedCandidateId":"local-6efe31bc4a0c0f37"},"review":{"state":"published-note","sourceScope":"full-text","availableScope":null,"readingCoverage":"selected-source-sections; not a complete section-by-section paper audit","completePaperAudit":false,"independentEvidenceReview":"pending-human-review","bilingualEditorialReview":"pending-human-review","authorship":"model-assisted-research-note","preparedFingerprint":"338453fb715fd5430af186141449e171ae3701c40029915174ce278397b53ffd"}},"sources":[{"id":"portbench-primary","title":"PortBench","url":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro","version":"2026-10-25 source capture","publicationDate":"2026-05-01","datePrecision":"day","type":"original-report","verifiedOn":"2026-10-25","authors":[],"documentId":"https://www.nist.gov/news-events/news/2026/05/caisi-evaluation-deepseek-v4-pro"}]}