WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← NUWA Index

Evaluation dossier

SEC-bench Pro

Tests long-horizon reproduction of working proof-of-concept inputs from disclosed vulnerability reports.

University of Illinois Urbana-Champaign / arXiv:2605.26548v2 / Capability

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

SEC-bench Pro · Headline success rate

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

24 observations · 16 dated

● Matched◇ Configuration differs□ Some conditions undisclosed
Headline success rate (%) GPT-5.5 [4] GPT-5.5 [1] GPT-5.4 [4] GPT-5.5 [2] GPT-5.5 [3] Opus 4.6 [4] GPT-5.4 [1] GPT-5.4 [2] Opus 4.6 [1] Opus 4.6 [3] GPT-5.4 [3] Opus 4.6 [2] 0 20 40 60 80 100 77.4% 58.4% 52.6% 47.6% 44.2% 39.4% 39% 35% 30.8% 27.9% 25% 22.3%
  1. 260617 / overall
  2. 260617 / v8
  3. 260617 / firefox
  4. 260617 / linux

8 observations lack a verified model release date. They remain in Data and Systems.

Select a point or table row to inspect its conditions and provenance.

24 observations

Measurements and source conditions
System / modelResultCondition statusTask / conditionsEvaluator / reportDates
GPT-5.577.4%Headline success rateIncomplete260617 / linuxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-04-23Report —
GPT-5.558.4%Headline success rateIncomplete260617 / overallheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-04-23Report —
GPT-5.452.6%Headline success rateIncomplete260617 / linuxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-03-05Report —
GPT-5.547.6%Headline success rateIncomplete260617 / v8headlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-04-23Report —
GPT-5.544.2%Headline success rateIncomplete260617 / firefoxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-04-23Report —
Opus 4.6Claude Opus 4.639.4%Headline success rateIncomplete260617 / linuxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.Claude CodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-05Report —
GPT-5.439%Headline success rateIncomplete260617 / overallheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-03-05Report —
GPT-5.435%Headline success rateIncomplete260617 / v8headlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-03-05Report —
Opus 4.6Claude Opus 4.630.8%Headline success rateIncomplete260617 / overallheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.Claude CodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-05Report —
Opus 4.6Claude Opus 4.627.9%Headline success rateIncomplete260617 / firefoxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.Claude CodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-05Report —
GPT-5.425%Headline success rateIncomplete260617 / firefoxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets.CodexNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-03-05Report —
Opus 4.6Claude Opus 4.622.3%Headline success rateIncomplete260617 / v8headlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.Claude CodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-05Report —
GLM-55.8%Headline success rateIncomplete260617 / firefoxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-12Report —
GLM-53.8%Headline success rateIncomplete260617 / overallheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-12Report —
GLM-53.6%Headline success rateIncomplete260617 / linuxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-12Report —
Kimi K2.52.9%Headline success rateIncomplete260617 / firefoxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
Kimi K2.52.3%Headline success rateIncomplete260617 / overallheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
Kimi K2.52.2%Headline success rateIncomplete260617 / linuxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
GLM-51.9%Headline success rateIncomplete260617 / v8headlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release 2026-02-12Report —
Kimi K2.51.9%Headline success rateIncomplete260617 / v8headlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
MiniMax M2.51.5%Headline success rateIncomplete260617 / linuxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
MiniMax M2.50.6%Headline success rateIncomplete260617 / overallheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
MiniMax M2.50%Headline success rateIncomplete260617 / v8headlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —
MiniMax M2.50%Headline success rateIncomplete260617 / firefoxheadlineHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets.OpenCodeNot disclosedSEC-bench Pro official leaderboard: selected 260617 and historical 260505Release —Report —

Evaluated by: Not disclosed
Reported by: SEC-bench Pro website

Task and measurement

Benchmark protocol profile

Task / subject
Coding agents reproducing vulnerability triggers in real software
Measurement target
Working proof-of-concept reproduction on a versioned software-security task set.
Scoring / outcome
Headline success rate · Source-reported measurement under the recorded configuration.
Interpretation boundary
A disclosed bug and supplied instructions constrain the task; grader choices and task-set version affect scores.

A reported evaluation

The source evaluates GPT-5.5 · 260617 / overall.

Run configuration: Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets..

Reported measurement: Headline success rate · 58.4%.

Source-reported measurement under the recorded configuration. 201/344

Headline success rate: 58.4 %. Conditions are retained from the source.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct
Can agents reproduce specific disclosed vulnerabilities in large security-critical codebases?
Task population
344 instances cover V8, SpiderMonkey and Linux; grading checks vulnerable, patched and latest counterparts and separates all-instance from completed-only rates.
Scoring and failure modes
All-instance success under final vulnerable/fixed/latest grading; completed-only rates are sensitivity analysis.
System conditions
344 instances across browser engines and Linux; model/scaffold timeout behavior differs.
Statistical meaning
Source-reported result statements; this does not establish trial counts, denominators or an estimator: The strongest reported model/scaffold solved 58% overall.
Validity and interpretation
Report-guided reproduction is not unguided discovery; target families and harnesses restrict external validity.
Appropriate research use
Model recommendation: use the all-instance denominator and fixed/latest evidence before interpreting reproduction gaps.

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

SEC-bench Pro website

SEC-bench Pro official leaderboard: selected 260617 and historical 260505 →

Actual evaluators: Not disclosed

48 result records · 6 reported system names

Protocols and comparison decisions (30)
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Completed-mode success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
SEC-bench Pro: Headline success rate · SEC-bench Pro websitesource-report · Scope reviewed
Original source charts and materials

Evaluation results

Reported by: SEC-bench Pro website · Executed by: Not established for every submission · Captured: 2026-10-26

Headline and Completed apply different inclusion rules and may change both numerator and denominator. Provider and backend labels do not identify the actual evaluator.

6 of 6 systems shown in source order

Headline success rate · 260617 / overall

SEC-bench ProHeadline success rate · 260617 / overallSource order · native units · configuration-specific0255075100GPT-5.5: 58.4%; 260617 / overall; Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.; 201/344GPT-5.558.4%GPT-5.4: 39%; 260617 / overall; Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.; 134/344GPT-5.439%Opus 4.6: 30.8%; 260617 / overall; Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.; 106/344Opus 4.630.8%GLM-5: 3.8%; 260617 / overall; Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.; 13/344GLM-53.8%Kimi K2.5: 2.3%; 260617 / overall; Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.; 8/344Kimi K2.52.3%MiniMax M2.5: 0.6%; 260617 / overall; Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets.; 2/344MiniMax M2.50.6%https://sec-bench.github.io/
  1. GPT-5.558.4%
    Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 201/344
  2. Codex · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 134/344
  3. Claude Code · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 106/344
  4. GLM-53.8%
    OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 13/344
  5. OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 8/344
  6. OpenCode · Headline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · 2/344
Source-reported systems and evaluation conditions
SystemResultTask set / modeConfigurationDates / attribution
GPT-5.558.4%201/344260617 / overallheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.560.3%199/330260617 / overallcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.439%134/344260617 / overallheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.440.5%132/326260617 / overallcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.630.8%106/344260617 / overallheadlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.672.8%99/136260617 / overallcompletedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-53.8%13/344260617 / overallheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-54.1%13/314260617 / overallcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.3%8/344260617 / overallheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.3%8/341260617 / overallcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50.6%2/344260617 / overallheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50.6%2/330260617 / overallcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=null; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.547.6%49/103260617 / v8headlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.548%49/102260617 / v8completedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.435%36/103260617 / v8headlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.436.7%36/98260617 / v8completedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.3h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.622.3%23/103260617 / v8headlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.661.1%22/36260617 / v8completedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-51.9%2/103260617 / v8headlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-52.4%2/84260617 / v8completedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.51.9%2/103260617 / v8headlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52%2/101260617 / v8completedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/103260617 / v8headlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/98260617 / v8completedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.544.2%46/104260617 / firefoxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.544.7%46/103260617 / firefoxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.425%26/104260617 / firefoxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.425%26/104260617 / firefoxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.627.9%29/104260617 / firefoxheadlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.658.1%25/43260617 / firefoxcompletedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-55.8%6/104260617 / firefoxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-56.2%6/96260617 / firefoxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.9%3/104260617 / firefoxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.9%3/103260617 / firefoxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/104260617 / firefoxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.50%0/100260617 / firefoxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.577.4%106/137260617 / linuxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.583.2%104/125260617 / linuxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.452.6%72/137260617 / linuxheadlineCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GPT-5.456.5%70/124260617 / linuxcompletedCodexHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.639.4%54/137260617 / linuxheadlineClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Opus 4.691.2%52/57260617 / linuxcompletedClaude CodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-53.6%5/137260617 / linuxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
GLM-53.7%5/134260617 / linuxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.2%3/137260617 / linuxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
Kimi K2.52.2%3/137260617 / linuxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=[]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.51.5%2/137260617 / linuxheadlineOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified
MiniMax M2.51.5%2/132260617 / linuxcompletedOpenCodeHeadline UI: fixed budget; Completed UI: within timeout. Published summary timeout_caps=["1.5h"]; cap labels do not establish equal token or cost budgets. · ? trials2026-06-17Source row date; semantics unspecified · SEC-bench Pro website · Executor: Not specified

Version and extraction scope →

Assessment provenance

SEC-bench Pro official snapshot → · External report

Related research questions

References

  1. [1] OpenAI — introducing gpt 5 5 ; Publication date and availability section; Parameter counts not disclosed in launch article
  2. [2] OpenAI — introducing gpt 5 4 ; Publication date and availability section; Parameter counts not disclosed in launch article
  3. [3] Claude Opus 4.6 — Anthropic ; Publication date and availability section; Official launch article; parameter counts not disclosed
  4. [4] Z.ai release notes ; GLM-5 entry dated 2026-02-12
  5. [5] glm-5 developer model card ; Introduction: model scaling
  6. [6] SEC-bench Pro official leaderboard: selected 260617 and historical 260505 ; /0/results/0/score_modes/headline/score; /0/results/0/score_modes/completed/score; /0/results/1/score_modes/headline/score
  7. [7] Hwiwon Lee ∗ Jiawei Liu  Dongjun Kim  Wubing Xia  Ziqi Zhang  Chunqiu Steven Xia  Lingming Zhang ∗. SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?. 2026-07-20. v2 [arXiv]; Author affiliations on the original paper title page; Source figure