Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models

Research question

How capable are language-model agents at multi-step security tasks, and which steps remain difficult when intermediate goals are supplied?

Design and assumptions

CTF tasks provide a description, local or remote starter material and an evaluator in containerized environments. Tasks span security categories and first-solve times. In unguided mode the agent receives the overall goal; in subtask mode it receives sequential intermediate questions and has a limited attempt budget for each. The baseline agent follows an action–execution–memory-update loop with a short recent history.

Source findings

Different models lead on different endpoints, so the source itself does not support a single ordering across unguided, guided and intermediate-task performance (§5.1).

The public MOTP example decomposes a web task into identifying relevant files, recognizing a condition and finishing the challenge. Intermediate answers make progress visible without guaranteeing independent completion (§§2.2–2.3, Table 1).

Scaffold variants and expanded memory change experimental conditions; the paper explicitly warns that stronger external evaluations use different iteration limits and scaffolds (Appendix I.1).

NUWA interpretation

Guidance measures the ability to execute supplied steps as well as independent discovery. First-solve time provides a difficulty proxy; it is not an individual human baseline directly comparable to METR task duration.

Limits and contrary evidence

Competition problems are intentionally solvable, typically short and small, and may not represent accidental vulnerabilities in large systems. Competition timing differs across events. Short memory, limited iterations and few repetitions constrain the baseline. Later Cybench subsets or harnesses require separate version labels.

Research connections

The source relates CTF challenges to cybersecurity capability evaluation. CyberGym substitutes real-project reproduction with a patched control, while AgentCyberRange changes the target to stateful range progression; no common denominator is established.

Research and decision relevance

Model recommendation: select the endpoint before comparing systems, display guidance and attempt rules, and avoid translating CTF completion into a probability of real-world compromise.