CyberGym

The page describes 1,507 instances from historical vulnerabilities in 188 software projects, derived from OSS-Fuzz.

Overall benchmark size does not establish any row denominator.

The stated reproduction criterion checks a result against the vulnerable and patched versions.

Scope is limited to the source conditions.

Success Rate is the proportion of instances with a successful reproduction; Trials counts attempts per instance, with any successful attempt solving the instance.

score_10 suffix does not define ten attempts. score_x1 is undefined.

The dynamic label denotes execution against a sanitized vulnerable environment or compiled binary. The test-time mem. label denotes knowledge or memory updated across instances; these are evaluation strategies.

Generic Memory and other raw feature tags must not be equated to test-time mem.

The page says individual teams evaluate and submit results; stochastic runs and ambiguous descriptions limit interpretation of modest score gaps.

This is a host caveat, not proof of each row executor or statistical significance.

The summary displays 34 zero-day vulnerabilities, 18 incomplete patches, 431 projects scanned and 969 average days undiscovered.

Aggregate scope is not reconciled with the detailed subsection totals.

The detailed reproduction analysis reports 759 generated inputs causing patched-version crashes across 60 projects; manual review identifies 17 incomplete patches spanning 15 projects, with none affecting latest releases.

These are high-level source-reported aggregate outcomes; no task artifacts or exploits extracted. Detailed count 17 differs from broader summary 18; scope of extra case is not explained here.

Latest-version validation reports 35 generated inputs with crashes, deduplicated into 10 previously unknown vulnerabilities, with mean persistence of 969 days.

No independent validation or operational artifacts included.

The open-ended study uses OpenHands with GPT-4.1 and GPT-5 over 431 projects containing 1,748 executables. Reported outcomes are 16 crashes and 7 confirmed zero-days for GPT-4.1, and 56 crashes and 22 confirmed zero-days for GPT-5, including 4 overlapping zero-days.

These counts use a different task than vulnerability reproduction; do not merge with leaderboard rates. Overlap with reproduction-derived findings is not fully specified; do not reconstruct the overall 34 total.

The page asserts a strong relationship between CyberGym performance and open-ended discovery capability.

The scoped webpage gives no correlation coefficient, uncertainty interval or matched-system analysis; qualitative claim only.

The prose reports a random subset of 300 tasks (~20%), comparing thinking modes for Qwen3-235B-A22B, GPT-5, Claude-3.7-Sonnet and Claude-Sonnet-4. GPT-5 success rises from 7.7% with minimal reasoning to 22.0% with high reasoning.

Subset comparison; do not substitute these percentages for full-dataset leaderboard fractions. Other model figure values not extracted.

The prose describes four difficulty levels, with additional information at levels 2 and 3. Level0 is reported at 3.5% reproduction without vulnerability text.

Level0 prose is rounded; retain separate original machine-data fraction, not a repaired value.

The prose associates longer ground-truth inputs with lower success; the [0, 10) byte range performs best. Inputs longer than 100 bytes represent 65.7% of the benchmark and have approximately 10% success.

Approximate ~10% remains approximate and is not promoted into an exact metric.

For OpenHands with Claude-4-Sonnet and maximum 100 steps, the prose concentrates successes in steps 20–80, peaking at 20–50; nearly half of runs end at 80–100 without success.

Ranges and nearly half are qualitative distribution descriptions, not exact success rates. Per-row leaderboard step budgets are not implied.

The page lists Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang and Dawn Song with UC Berkeley affiliation, and cites the work at ICLR 2026.

No precise publication or revision date established by this citation.