NUWA Evidence record
Kimi K2.5
SEC-bench Pro · Evidence record
Source snapshot, chart and review scope →
Original source charts and materials
Recorded measurements
- Headline success rate
- 1.9%
2/103
OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.
Method and scope
Working proof-of-concept reproduction on a versioned software-security task set.
- Evidence type
- Benchmark
- Assessment subject
- Coding agents reproducing vulnerability triggers in real software
- Primary dimension
- Capability
- Evidence snapshot
- 2026-10-29.1
Primary measure is successful task completion under a specified agent setup.
Source finding
Headline success rate: 1.9 %. Conditions are retained from the source.
Limitations
A disclosed bug and supplied instructions constrain the task; grader choices and task-set version affect scores.
Headline and Completed have mode-specific numerators and denominators. Provider and serving backend do not identify the actual evaluator. No universal time cap is inferred; inspect source configuration. Official source-reported values; no experimental replication. Model provider and backend are not evidence of per-row evaluator/executor identity. Row date has unspecified semantics; snapshot ID and retrieval time are separate. Source verified:false retained; independent verification semantics not established.
Original figure retained without estimating numerical values. Full source tables have not been transcribed.
Related research questions
Cybersecurity / Vulnerability reproduction →
The paper evaluates working reproductions of disclosed bugs.
Cybersecurity / Cyber task execution →
Vulnerability reproduction informs cyber task capability but not a full intrusion chain.
References
- [1] Hwiwon Lee ∗ Jiawei Liu Dongjun Kim Wubing Xia Ziqi Zhang Chunqiu Steven Xia Lingming Zhang ∗. SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?. 2026-07-20. v2 [arXiv]; Author affiliations on the original paper title page; Source figure
- [2] SEC-bench Pro official leaderboard: selected 260617 and historical 260505 ; /1/results/4/score_modes/headline/score