WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← NUWA Index

Source record

SEC-bench Pro

Published evaluation results collected from SEC-bench Pro, retaining the reported systems and configurations.

Original source and provenance ↓

Evaluation results

Reported by: SEC-bench Pro website · Executed by: Not established for every submission · Captured: 2026-10-26

Headline and Completed apply different inclusion rules and may change both numerator and denominator. Provider and backend labels do not identify the actual evaluator.

6 of 6 systems shown in source order

Headline success rate · 260617 / overall

  1. GPT-5.558.4%
    Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 201/344
  2. Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 134/344
  3. Claude Code · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 106/344
  4. GLM-53.8%
    OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 13/344
  5. OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 8/344
  6. OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 2/344
Source-reported systems and evaluation conditions
SystemResultTask set / modeConfigurationDates / attribution
GPT-5.558.4%201/344260617 / overallheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.560.3%199/330260617 / overallcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.439%134/344260617 / overallheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.440.5%132/326260617 / overallcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.630.8%106/344260617 / overallheadlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.672.8%99/136260617 / overallcompletedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-53.8%13/344260617 / overallheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-54.1%13/314260617 / overallcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.3%8/344260617 / overallheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.3%8/341260617 / overallcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50.6%2/344260617 / overallheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50.6%2/330260617 / overallcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.547.6%49/103260617 / v8headlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.548%49/102260617 / v8completedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.435%36/103260617 / v8headlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.436.7%36/98260617 / v8completedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.622.3%23/103260617 / v8headlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.661.1%22/36260617 / v8completedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-51.9%2/103260617 / v8headlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-52.4%2/84260617 / v8completedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.51.9%2/103260617 / v8headlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52%2/101260617 / v8completedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/103260617 / v8headlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/98260617 / v8completedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.544.2%46/104260617 / firefoxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.544.7%46/103260617 / firefoxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.425%26/104260617 / firefoxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.425%26/104260617 / firefoxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.627.9%29/104260617 / firefoxheadlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.658.1%25/43260617 / firefoxcompletedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-55.8%6/104260617 / firefoxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-56.2%6/96260617 / firefoxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.9%3/104260617 / firefoxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.9%3/103260617 / firefoxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/104260617 / firefoxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/100260617 / firefoxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.577.4%106/137260617 / linuxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.583.2%104/125260617 / linuxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.452.6%72/137260617 / linuxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.456.5%70/124260617 / linuxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.639.4%54/137260617 / linuxheadlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.691.2%52/57260617 / linuxcompletedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-53.6%5/137260617 / linuxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-53.7%5/134260617 / linuxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.2%3/137260617 / linuxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.2%3/137260617 / linuxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.51.5%2/137260617 / linuxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.51.5%2/132260617 / linuxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified

Scope and attribution

Reported by
SEC-bench Pro website
Actual evaluator
Not established for every observation
Captured version
2026-10-26
Source identity
a2c735df11620c32

Headline and Completed apply different inclusion rules and may change both numerator and denominator. Provider and backend labels do not identify the actual evaluator.

Source findings

The following statements summarize the source. They are not independent NUWA experimental findings.

Selected snapshot 260617 has 344 Overall instances: V8 103, Firefox 104, Linux 137.

official-root embedded /0/instances,/1/instances,/2/instances,/3/instances

Source-reported; no replication.

Historical snapshot 260505 has 183 Overall instances: V8 103 and Firefox 80; no Linux track.

official-260505 embedded /0/instances,/1/instances,/2/instances and selected version option

Source-reported; no replication.

Headline includes all instances; Completed excludes instances with trajectory timeout markers, including any successes on those instances.

official-v8-opus-detail.html:97-138; official-root embedded /0/results/2/score_modes

Source-reported; no replication.

The published Provider column comes from org, while Backend and agent scaffold are distinct fields. Per-row evaluator identity is not disclosed.

official-root.html Pro table headers; embedded /0/results/*/org,/backend,/agent

Source-reported; no replication.

GPT-5.4 V8 summary timeout_caps is ["1.3h"], while most nonempty arrays say ["1.5h"].

official-root embedded /1/results/1/details_summary/timeout_caps

Source-reported; no replication.

Publisher reports newer Claude Opus and Mythos results unavailable due to safeguard restrictions.

official-root embedded /0/footnotes/0

Source-reported; no replication.

Pro News announces Overall/Linux June 17 2026, SpiderMonkey May 5 2026, and V8 launch May 1 2026. These are announcement dates, separate from row dates and capture timestamps.

official-root Pro info_sections/News

Source-reported; no replication.

The renderer defaults to headline, selects mode-specific scores and ranks, and retains source rank labels while sorting visible rows. Historical missing-target version controls fall back to Overall.

official-renderer.js:6,63-111,211-259,280-344

Source-reported; no replication.

Extraction coverage

Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.

View 14 coverage entries
SectionTreatmentLocation and scope
SEC-bench Pro / snapshot 260617 / overallextractedhttps://sec-bench.github.io/ script#leaderboard-data /0All 6 source rows and both scoring modes extracted; snapshot instance count 344; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
SEC-bench Pro / snapshot 260617 / v8extractedhttps://sec-bench.github.io/ script#leaderboard-data /1All 6 source rows and both scoring modes extracted; snapshot instance count 103; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
SEC-bench Pro / snapshot 260617 / firefoxextractedhttps://sec-bench.github.io/ script#leaderboard-data /2All 6 source rows and both scoring modes extracted; snapshot instance count 104; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
SEC-bench Pro / snapshot 260617 / linuxextractedhttps://sec-bench.github.io/ script#leaderboard-data /3All 6 source rows and both scoring modes extracted; snapshot instance count 137; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
SEC-bench Pro / snapshot 260505 / overallextractedhttps://sec-bench.github.io/260505/ script#leaderboard-data /0All 6 source rows and both scoring modes extracted; snapshot instance count 183; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
SEC-bench Pro / snapshot 260505 / v8reusedhttps://sec-bench.github.io/260505/ script#leaderboard-data /1All 6 source rows and both scoring modes extracted; snapshot instance count 103; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
SEC-bench Pro / snapshot 260505 / firefoxextractedhttps://sec-bench.github.io/260505/ script#leaderboard-data /2All 6 source rows and both scoring modes extracted; snapshot instance count 80; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result.
Original SEC-bench classic tablesnot-relevantofficial-root.html [data-mode-panel="classic"]; embedded /4 and /5Original PoC Generation and Vulnerability Patching tables excluded; their mode definitions, dates and scores do not apply to Pro.
Renderer semanticsextractedofficial-renderer.js:6,63-111,211-259,280-344Headline default; mode switch reads mode-specific score/rank/count, changes bar titles/widths, and re-sorts. Rank labels retain source mode rank. Missing selected target falls back to Overall, not a new target result. Presentation widths are not benchmark metrics.
Headline and Completed definitionsqualitativeofficial-v8-opus-detail.html:97-138Detail labels all instances vs exclude timeout markers; timeout uses per-instance marker under trajectory artifact. Only definition and aggregate context used, no per-instance extraction.
Safeguard-restricted newer Claude Opus/Mythos rowsunavailableofficial-root embedded /0/footnotes/0 (repeated on each Pro track)Publisher states newer results unavailable due to safeguards. No numeric score assigned, no versions invented.
Executor, exact scaffold versions and universal budgetnot-disclosedAll Pro results; details_summary timeout_capsExecutor/evaluator identity and exact agent/backend software versions absent. Overall has no timeout-cap summary. No equal-token/equal-cost or universal 1.5h assertion.
Snapshot identifiers, row date and publication metadataextractedofficial-root.html version options and Pro News; all Pro /results/*/date; capture-log.jsonBoth snapshots rows date 2026-06-17; historical 260505 is a selectable snapshot ID, not proof of evaluation date. HTTP Last-Modified 2026-07-09 is transport metadata, not asserted as publication/revision date.
Per-instance runs, trajectories, exploit tasks, papernot-relevantScope boundaryOnly leaderboard aggregates, exact embedded summary fields, renderer and mode definition extracted. No per-instance operational exploit content or external paper result extraction.

Unresolved source issues

Headline 与 Completed 使用不同样本集合,分子和分母都可能改变,不能直接将分数差异解释为能力变化。

来源未披露各条结果的实际执行方;日期标签与快照编号不代表已核实的实验时间。

Version fingerprints
document

https://sec-bench.github.io/

e01142eb1af97dfc56ba68467ac87971a9f2377e922cc33bf34b1bcc679a90e5
renderer

https://sec-bench.github.io/js/leaderboard.js?v=6

a5f2e7a0f2eb04e4fc1da6eb41d08b7ffb91d3f4a984ba0044afda9bd3be9f46
document

https://sec-bench.github.io/260505/

53f9744c15a073f287298fa221028fbac9ac137ea2d0f7f2b037adb85babf9e3
document

https://sec-bench.github.io/v8/runs/claude_opus-4.6-max/

4a71cc599862afdf583b366563ccccea8f2a4b30b77f08f8bfff8fe99c276e2e
data

https://sec-bench.github.io/#leaderboard-data

6d2e9bdd9cbe11bca1b7e2e490f18fc3f7fc251f374a119d54d4ac8ac5189708
data

https://sec-bench.github.io/260505/#leaderboard-data

e16da95f7fb33b8fd3b74400235567f28e7ba088ef9f2a29f7c16b8806156c07

References

Numerical results are reported by the cited authors.

  1. [1] SEC-bench Pro official leaderboard: selected 260617 and historical 260505