CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

Research question

Can agents reproduce real historical software vulnerabilities at scale, and how do input information, model and agent configuration change that ability?

Design and assumptions

Historical OSS-Fuzz cases are filtered for informative descriptions, reproducibility and redundancy. Agents receive a codebase and executable, iteratively submit candidate inputs and observe execution feedback. The primary level supplies a vulnerability description; other levels remove it or add a stack trace and patch information. The study focuses on sanitizer-detectable memory-safety failures, chiefly in C/C++ projects.

Source findings

The difficulty ladder demonstrates that richer diagnostic information makes reproduction easier for the examined configuration (§4, Figure 6).

Pre- and post-training-cutoff splits do not show a statistically significant success-rate difference for the examined models (Table 2). This is limited evidence against one contamination effect, not proof of an uncontaminated benchmark.

The patched-version control helps distinguish a target-specific reproducer from an unrelated crash; the authors separately investigate incomplete patches and new vulnerabilities (§§3.2, 5).

NUWA interpretation

Execution-based validation gives the score a concrete meaning. Reproduction capability nevertheless differs from reliable exploitation, choosing malicious goals or progressing through a network. New-vulnerability investigations are additional experiments rather than replacements for the primary benchmark endpoint.

Limits and contrary evidence

Coverage favors sanitizer-visible memory safety and the available OSS-Fuzz/ARVO corpus. Description filtering and rephrasing use an LLM and a sampled human quality check. Public code and reports create contamination uncertainty. More input information changes the task. A later report’s different budget or scaffold cannot be treated as a controlled model-only improvement.

Research connections

ARVO supplies much of the reproducible case infrastructure (§3.3). Cybench uses competition tasks and flags; AgentCyberRange examines state-dependent progression through a range. These are complementary constructs, not interchangeable score scales.

Research and decision relevance

Model recommendation: require both vulnerable-trigger and patched-clean receipts, identify the exact task subset and information level, and keep run budget and agent settings alongside each score.