Cybench

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct

How capable are language-model agents at multi-step security tasks, and which steps remain difficult when intermediate goals are supplied?

Task population

CTF tasks provide a description, local or remote starter material and an evaluator in containerized environments. Tasks span security categories and first-solve times. In unguided mode the agent receives the overall goal; in subtask mode it receives sequential intermediate questions and has a limited attempt budget for each. The baseline agent follows an action–execution–memory-update loop with a short recent history.

Scoring and failure modes

Unguided success and final guided success are binary task endpoints; subtask performance is the fraction of intermediate questions solved, macro-averaged across tasks. Table 2 gives single attempts; Table 3 uses the maximum of multiple attempts. A unique success marker or correct submitted answer determines completion.

System conditions

CTF tasks provide a description, local or remote starter material and an evaluator in containerized environments. Tasks span security categories and first-solve times. In unguided mode the agent receives the overall goal; in subtask mode it receives sequential intermediate questions and has a limited attempt budget for each. The baseline agent follows an action–execution–memory-update loop with a short recent history.

Statistical meaning

Table 2 is a single-attempt evaluation; Table 3 takes the best of three attempts. Subtask metrics are macro-averaged, not a pooled count of all questions. Unreported interval endpoints remain unknown.

Validity and interpretation

Competition problems are intentionally solvable, typically short and small, and may not represent accidental vulnerabilities in large systems. Competition timing differs across events. Short memory, limited iterations and few repetitions constrain the baseline. Later Cybench subsets or harnesses require separate version labels.

Appropriate research use

Model recommendation: select the endpoint before comparing systems, display guidance and attempt rules, and avoid translating CTF completion into a probability of real-world compromise.

• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · §2.1 Task SpecificationarXiv:2408.08926v4
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · §2.2 Task Example: MOTP; §2.3 Subtasks; Table 1arXiv:2408.08926v4
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · §2.4 MetricsarXiv:2408.08926v4
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · §3.3 Task Selection; Appendix F First Solve TimearXiv:2408.08926v4
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · §4 LM-based Agent; §5.1 Model Capabilities; Tables 2–3arXiv:2408.08926v4
• Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models · Appendix I.1 Limited agent scaffolding; I.2 Limitation of data sourcesarXiv:2408.08926v4