WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← NUWA Index

Evidence record

Llama 3.1 405B Instruct

Cybench · Evidence record

Stanford University / arXiv:2408.08926v4 / Capability

Sources and provenance ↓ Read the research narrative →

Original source charts and materials

Reported results

Source order · reported configurations

Unguided performance

0–100%
Llama 3.1 405B Instruct
7.5%
Showing 1 of 1 systems

Values describe the measured outcome under the source protocol, not a general safety score.

Table 2: one attempt per task with the structured bash agent. Unguided performance is averaged across tasks; guided and subtask performance are macro-averaged. Evaluation date not specified; publication / record dates are shown below.

Evaluation conditions and system configurations
Llama 3.1 405B Instruct
40 CTF tasks · Table 2 · Cybench
modelVersion: not specified · harness: Structured bash · tools: not specified · budget: not specified
Reported results · source order
System / configurationUnguided performanceSubtask-guided performanceSubtask performancePublication / verification
Llama 3.1 405B InstructStructured bash7.5%15.0%20.5%Source checked2026-10-25

Method and scope

Task completion on 40 professional CTF problems.

Evidence type
Benchmark
Assessment subject
AI agent solving CTF tasks
Primary dimension
Capability
Evidence snapshot
2026-10-29.1

Primary dimension follows the source’s measured construct. Metric-level interpretations may differ.

Source finding

Source-reported measurements for Llama 3.1 405B Instruct in Table 2; values are transcribed, not independently reproduced.

Limitations

Competition tasks and supplied guidance differ from operational attacks.

Configuration, task and aggregation differences remain as reported. Source order is retained. Transcription is not independent replication; no cross-protocol ranking is supported.

Related research questions

References

Numerical results are reported by the cited authors.

  1. [1] Andy K. Zhang, Neil Perry, et al.. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. 2025-04-12. v4 [arXiv]; Author affiliations on the original paper title page; Table 2