CyberGym

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

Can agents reproduce real historical software vulnerabilities at scale, and how do input information, model and agent configuration change that ability?

Task population

Historical OSS-Fuzz cases are filtered for informative descriptions, reproducibility and redundancy. Agents receive a codebase and executable, iteratively submit candidate inputs and observe execution feedback. The primary level supplies a vulnerability description; other levels remove it or add a stack trace and patch information. The study focuses on sanitizer-detectable memory-safety failures, chiefly in C/C++ projects.

Scoring and failure modes

A successful reproducer triggers a sanitizer failure in the vulnerable version and no sanitizer failure in the patched version. Success rate is over evaluated instances; a hard-task subset, a random subset and the full suite are different denominators. Agent feedback and final receipt must be distinguished.

System conditions

Historical OSS-Fuzz cases are filtered for informative descriptions, reproducibility and redundancy. Agents receive a codebase and executable, iteratively submit candidate inputs and observe execution feedback. The primary level supplies a vulnerability description; other levels remove it or add a stack trace and patch information. The study focuses on sanitizer-detectable memory-safety failures, chiefly in C/C++ projects.

Statistical meaning

§4 Table 2 retains success numerators/denominators and Fisher exact / two-proportion Z tests for cutoff splits. Non-significance is not equivalence. Other result-specific intervals remain in their source records.

Validity and interpretation

Coverage favors sanitizer-visible memory safety and the available OSS-Fuzz/ARVO corpus. Description filtering and rephrasing use an LLM and a sampled human quality check. Public code and reports create contamination uncertainty. More input information changes the task. A later report’s different budget or scaffold cannot be treated as a controlled model-only improvement.

Appropriate research use

Model recommendation: require both vulnerable-trigger and patched-clean receipts, identify the exact task subset and information level, and keep run budget and agent settings alongside each score.

• CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · §3.1 Scope, Rationale, and LimitationsarXiv:2506.02548v3
• CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · §3.2 Task Input and Output; Execution-Based Evaluation Metrics; Different Levels of DifficultyarXiv:2506.02548v3
• CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · §3.3 Quality Assurance; Benchmark Scale and Diversity; Table 3arXiv:2506.02548v3
• CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · §4 Limited Impact of Potential Data Contamination; Table 2arXiv:2506.02548v3
• CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · §4 Richer Input Information Enhances Reproduction Effort; Figure 6arXiv:2506.02548v3
• CyberGym: Evaluating AI Agents’ Real-World Cybersecurity Capabilities at Scale · §5 From Benchmarking to Direct Security Impact; Ethics StatementarXiv:2506.02548v3