NUWA Choose the right measurement
Compare 2–4 protocol profiles before interpreting or combining their results. Unknown methods remain explicit gaps.
Measurement profiles; heterogeneous scores are not ranked or pooled.
| Measurement design | CyberGymarXiv:2506.02548v3 | CybencharXiv:2408.08926v4 | AgentCyberRangeWebsite bundle routes-B-0Mubrq.js / 2026-10-24 capture |
|---|---|---|---|
| Reading and editorial scope | Published model-assisted note · independent review pending | Published model-assisted note · independent review pending | Published model-assisted note · independent review pending |
| Measurement target | Can agents reproduce real historical software vulnerabilities at scale, and how do input information, model and agent configuration change that ability? | How capable are language-model agents at multi-step security tasks, and which steps remain difficult when intermediate goals are supplied? | Can an agent discover an attack surface from sparse entry information and continue making decisions as access and information accumulate? |
| Tasks and population | Historical OSS-Fuzz cases are filtered for informative descriptions, reproducibility and redundancy. Agents receive a codebase and executable, iteratively submit candidate inputs and observe execution feedback. The primary level supplies a vulnerability description; other levels remove it or add a stack trace and patch information. The study focuses on sanitizer-detectable memory-safety failures, chiefly in C/C++ projects. | CTF tasks provide a description, local or remote starter material and an evaluator in containerized environments. Tasks span security categories and first-solve times. In unguided mode the agent receives the overall goal; in subtask mode it receives sequential intermediate questions and has a limited attempt budget for each. The baseline agent follows an action–execution–memory-update loop with a short recent history. | The team introduction starts agents from entry IP information and describes progression through linked environments. Earlier access and discovered clues change later options. Web and post-access tracks are presented separately. The trajectory figure retains unsuccessful exploration as well as successful progress, illustrating a process rather than just a final endpoint. |
| Scoring and success | A successful reproducer triggers a sanitizer failure in the vulnerable version and no sanitizer failure in the patched version. Success rate is over evaluated instances; a hard-task subset, a random subset and the full suite are different denominators. Agent feedback and final receipt must be distinguished. | Unguided success and final guided success are binary task endpoints; subtask performance is the fraction of intermediate questions solved, macro-averaged across tasks. Table 2 gives single attempts; Table 3 uses the maximum of multiple attempts. A unique success marker or correct submitted answer determines completion. | The existing dated catalogue preserves the official site’s Level-0 Pass@3 labels. The fetched introduction does not independently establish the pass@3 estimator, per-cell trial denominators, complete budgets or all submission settings. Those fields remain unknown in this new profile. |
| System and resource conditions | Historical OSS-Fuzz cases are filtered for informative descriptions, reproducibility and redundancy. Agents receive a codebase and executable, iteratively submit candidate inputs and observe execution feedback. The primary level supplies a vulnerability description; other levels remove it or add a stack trace and patch information. The study focuses on sanitizer-detectable memory-safety failures, chiefly in C/C++ projects. | CTF tasks provide a description, local or remote starter material and an evaluator in containerized environments. Tasks span security categories and first-solve times. In unguided mode the agent receives the overall goal; in subtask mode it receives sequential intermediate questions and has a limited attempt budget for each. The baseline agent follows an action–execution–memory-update loop with a short recent history. | The team introduction starts agents from entry IP information and describes progression through linked environments. Earlier access and discovered clues change later options. Web and post-access tracks are presented separately. The trajectory figure retains unsuccessful exploration as well as successful progress, illustrating a process rather than just a final endpoint. |
| Statistical meaning | §4 Table 2 retains success numerators/denominators and Fisher exact / two-proportion Z tests for cutoff splits. Non-significance is not equivalence. Other result-specific intervals remain in their source records. | Table 2 is a single-attempt evaluation; Table 3 takes the best of three attempts. Subtask metrics are macro-averaged, not a pooled count of all questions. Unreported interval endpoints remain unknown. | Not established by the retained material |
| Validity and limitations | Coverage favors sanitizer-visible memory safety and the available OSS-Fuzz/ARVO corpus. Description filtering and rephrasing use an LLM and a sampled human quality check. Public code and reports create contamination uncertainty. More input information changes the task. A later report’s different budget or scaffold cannot be treated as a controlled model-only improvement. | Competition problems are intentionally solvable, typically short and small, and may not represent accidental vulnerabilities in large systems. Competition timing differs across events. Short memory, limited iterations and few repetitions constrain the baseline. Later Cybench subsets or harnesses require separate version labels. | The cached project page is a JavaScript shell. The accessible research introduction omits complete scoring, budget, model-release identity and independent-run counts. A trajectory figure cannot establish a population rate. The introduction’s application requests do not prove deployments or partnerships. |
| Appropriate research use | Model recommendation: require both vulnerable-trigger and patched-clean receipts, identify the exact task subset and information level, and keep run budget and agent settings alongside each score. | Model recommendation: select the endpoint before comparing systems, display guidance and attempt rules, and avoid translating CTF completion into a probability of real-world compromise. | Model recommendation: obtain the archived benchmark specification and submission configuration before interpreting model gaps; use the introduction for task scope and process reading only. |