Source record
SEC-bench Pro
Published evaluation results collected from SEC-bench Pro, retaining the reported systems and configurations.
Evaluation results
Filter systems and configurations
Reported by: SEC-bench Pro website · Executed by: Not established for every submission · Captured: 2026-10-26
Headline and Completed apply different inclusion rules and may change both numerator and denominator. Provider and backend labels do not identify the actual evaluator.
6 of 6 systems shown in source order
Headline success rate · 260617 / overall
GPT-5.558.4%Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 201/344
GPT-5.439%Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 134/344
Opus 4.630.8%Claude Code · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 106/344
GLM-53.8%OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 13/344
Kimi K2.52.3%OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 8/344
MiniMax M2.50.6%OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 2/344
| System | Result | Task set / mode | Configuration | Dates / attribution |
|---|---|---|---|---|
| 58.4%201/344 | 260617 / overallheadline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 60.3%199/330 | 260617 / overallcompleted | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 39%134/344 | 260617 / overallheadline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 40.5%132/326 | 260617 / overallcompleted | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 30.8%106/344 | 260617 / overallheadline | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 72.8%99/136 | 260617 / overallcompleted | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 3.8%13/344 | 260617 / overallheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 4.1%13/314 | 260617 / overallcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.3%8/344 | 260617 / overallheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.3%8/341 | 260617 / overallcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 0.6%2/344 | 260617 / overallheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 0.6%2/330 | 260617 / overallcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 47.6%49/103 | 260617 / v8headline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 48%49/102 | 260617 / v8completed | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 35%36/103 | 260617 / v8headline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 36.7%36/98 | 260617 / v8completed | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 22.3%23/103 | 260617 / v8headline | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 61.1%22/36 | 260617 / v8completed | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 1.9%2/103 | 260617 / v8headline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.4%2/84 | 260617 / v8completed | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 1.9%2/103 | 260617 / v8headline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2%2/101 | 260617 / v8completed | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 0%0/103 | 260617 / v8headline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 0%0/98 | 260617 / v8completed | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 44.2%46/104 | 260617 / firefoxheadline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 44.7%46/103 | 260617 / firefoxcompleted | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 25%26/104 | 260617 / firefoxheadline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 25%26/104 | 260617 / firefoxcompleted | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 27.9%29/104 | 260617 / firefoxheadline | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 58.1%25/43 | 260617 / firefoxcompleted | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 5.8%6/104 | 260617 / firefoxheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 6.2%6/96 | 260617 / firefoxcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.9%3/104 | 260617 / firefoxheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.9%3/103 | 260617 / firefoxcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 0%0/104 | 260617 / firefoxheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 0%0/100 | 260617 / firefoxcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 77.4%106/137 | 260617 / linuxheadline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 83.2%104/125 | 260617 / linuxcompleted | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 52.6%72/137 | 260617 / linuxheadline | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 56.5%70/124 | 260617 / linuxcompleted | CodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 39.4%54/137 | 260617 / linuxheadline | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 91.2%52/57 | 260617 / linuxcompleted | Claude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 3.6%5/137 | 260617 / linuxheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 3.7%5/134 | 260617 / linuxcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.2%3/137 | 260617 / linuxheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 2.2%3/137 | 260617 / linuxcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 1.5%2/137 | 260617 / linuxheadline | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified | |
| 1.5%2/132 | 260617 / linuxcompleted | OpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials | 2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified |
Scope and attribution
- Reported by
- SEC-bench Pro website
- Actual evaluator
- Not established for every observation
- Captured version
- 2026-10-26
- Source identity
- a2c735df11620c32
Headline and Completed apply different inclusion rules and may change both numerator and denominator. Provider and backend labels do not identify the actual evaluator.
Source findings
The following statements summarize the source. They are not independent NUWA experimental findings.
Selected snapshot 260617 has 344 Overall instances: V8 103, Firefox 104, Linux 137.
official-root embedded /0/instances,/1/instances,/2/instances,/3/instancesSource-reported; no replication.
Historical snapshot 260505 has 183 Overall instances: V8 103 and Firefox 80; no Linux track.
official-260505 embedded /0/instances,/1/instances,/2/instances and selected version optionSource-reported; no replication.
Headline includes all instances; Completed excludes instances with trajectory timeout markers, including any successes on those instances.
official-v8-opus-detail.html:97-138; official-root embedded /0/results/2/score_modesSource-reported; no replication.
The published Provider column comes from org, while Backend and agent scaffold are distinct fields. Per-row evaluator identity is not disclosed.
official-root.html Pro table headers; embedded /0/results/*/org,/backend,/agentSource-reported; no replication.
GPT-5.4 V8 summary timeout_caps is ["1.3h"], while most nonempty arrays say ["1.5h"].
official-root embedded /1/results/1/details_summary/timeout_capsSource-reported; no replication.
Publisher reports newer Claude Opus and Mythos results unavailable due to safeguard restrictions.
official-root embedded /0/footnotes/0Source-reported; no replication.
Pro News announces Overall/Linux June 17 2026, SpiderMonkey May 5 2026, and V8 launch May 1 2026. These are announcement dates, separate from row dates and capture timestamps.
official-root Pro info_sections/NewsSource-reported; no replication.
The renderer defaults to headline, selects mode-specific scores and ranks, and retains source rank labels while sorting visible rows. Historical missing-target version controls fall back to Overall.
official-renderer.js:6,63-111,211-259,280-344Source-reported; no replication.
Extraction coverage
Each entry records the treatment of the source section. A reviewed observation does not imply that every claim or result in the document was verified.
View 14 coverage entries
| Section | Treatment | Location and scope |
|---|---|---|
| SEC-bench Pro / snapshot 260617 / overall | extracted | https://sec-bench.github.io/ script#leaderboard-data /0All 6 source rows and both scoring modes extracted; snapshot instance count 344; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| SEC-bench Pro / snapshot 260617 / v8 | extracted | https://sec-bench.github.io/ script#leaderboard-data /1All 6 source rows and both scoring modes extracted; snapshot instance count 103; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| SEC-bench Pro / snapshot 260617 / firefox | extracted | https://sec-bench.github.io/ script#leaderboard-data /2All 6 source rows and both scoring modes extracted; snapshot instance count 104; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| SEC-bench Pro / snapshot 260617 / linux | extracted | https://sec-bench.github.io/ script#leaderboard-data /3All 6 source rows and both scoring modes extracted; snapshot instance count 137; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| SEC-bench Pro / snapshot 260505 / overall | extracted | https://sec-bench.github.io/260505/ script#leaderboard-data /0All 6 source rows and both scoring modes extracted; snapshot instance count 183; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| SEC-bench Pro / snapshot 260505 / v8 | reused | https://sec-bench.github.io/260505/ script#leaderboard-data /1All 6 source rows and both scoring modes extracted; snapshot instance count 103; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| SEC-bench Pro / snapshot 260505 / firefox | extracted | https://sec-bench.github.io/260505/ script#leaderboard-data /2All 6 source rows and both scoring modes extracted; snapshot instance count 80; all numeric details_summary fields retained independently. Historical snapshot is not an independently captured historical-day result. |
| Original SEC-bench classic tables | not-relevant | official-root.html [data-mode-panel="classic"]; embedded /4 and /5Original PoC Generation and Vulnerability Patching tables excluded; their mode definitions, dates and scores do not apply to Pro. |
| Renderer semantics | extracted | official-renderer.js:6,63-111,211-259,280-344Headline default; mode switch reads mode-specific score/rank/count, changes bar titles/widths, and re-sorts. Rank labels retain source mode rank. Missing selected target falls back to Overall, not a new target result. Presentation widths are not benchmark metrics. |
| Headline and Completed definitions | qualitative | official-v8-opus-detail.html:97-138Detail labels all instances vs exclude timeout markers; timeout uses per-instance marker under trajectory artifact. Only definition and aggregate context used, no per-instance extraction. |
| Safeguard-restricted newer Claude Opus/Mythos rows | unavailable | official-root embedded /0/footnotes/0 (repeated on each Pro track)Publisher states newer results unavailable due to safeguards. No numeric score assigned, no versions invented. |
| Executor, exact scaffold versions and universal budget | not-disclosed | All Pro results; details_summary timeout_capsExecutor/evaluator identity and exact agent/backend software versions absent. Overall has no timeout-cap summary. No equal-token/equal-cost or universal 1.5h assertion. |
| Snapshot identifiers, row date and publication metadata | extracted | official-root.html version options and Pro News; all Pro /results/*/date; capture-log.jsonBoth snapshots rows date 2026-06-17; historical 260505 is a selectable snapshot ID, not proof of evaluation date. HTTP Last-Modified 2026-07-09 is transport metadata, not asserted as publication/revision date. |
| Per-instance runs, trajectories, exploit tasks, paper | not-relevant | Scope boundaryOnly leaderboard aggregates, exact embedded summary fields, renderer and mode definition extracted. No per-instance operational exploit content or external paper result extraction. |
Unresolved source issues
Headline 与 Completed 使用不同样本集合,分子和分母都可能改变,不能直接将分数差异解释为能力变化。
来源未披露各条结果的实际执行方;日期标签与快照编号不代表已核实的实验时间。
Version fingerprints
https://sec-bench.github.io/
e01142eb1af97dfc56ba68467ac87971a9f2377e922cc33bf34b1bcc679a90e5https://sec-bench.github.io/js/leaderboard.js?v=6
a5f2e7a0f2eb04e4fc1da6eb41d08b7ffb91d3f4a984ba0044afda9bd3be9f46https://sec-bench.github.io/260505/
53f9744c15a073f287298fa221028fbac9ac137ea2d0f7f2b037adb85babf9e3https://sec-bench.github.io/v8/runs/claude_opus-4.6-max/
4a71cc599862afdf583b366563ccccea8f2a4b30b77f08f8bfff8fe99c276e2ehttps://sec-bench.github.io/#leaderboard-data
6d2e9bdd9cbe11bca1b7e2e490f18fc3f7fc251f374a119d54d4ac8ac5189708https://sec-bench.github.io/260505/#leaderboard-data
e16da95f7fb33b8fd3b74400235567f28e7ba088ef9f2a29f7c16b8806156c07References
Numerical results are reported by the cited authors.