NUWA Evaluation dossier
Meta CyberSecEval / CyberSecEval 4
Provides evaluation suites for cyber capabilities and safeguards, with distinct task families and release-specific protocols.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system under the original task configuration
- Measurement target
- Provides evaluation suites for cyber capabilities and safeguards, with distinct task families and release-specific protocols.
- Interpretation boundary
- Results from different suite generations and task families require separate interpretation.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- What security constructs does the CyberSecEval suite cover?
- Task population
- The version-four documentation separates offensive assistance, injection/interpreter risks and defensive tasks, adding CyberSOCEval and AutoPatchBench.
- Scoring and failure modes
- Not established by the retained material
- System conditions
- Not established by the retained material
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- The introductory documentation does not establish all component protocols, versions, sample counts or comparable scores.
- Appropriate research use
- Model recommendation: select and cite the component-specific documentation before comparing models.
- Introduction · Introduction: CyberSecEval 4 overviewSelected original resource https://meta-llama.github.io/PurpleLlama/CyberSecEval/docs/intro; sha256:d1fa0b20e03bb8672ef79d157f38f9167109068cfcf0ad7f12ada2abcc3457f7; archived source capture; exact capture time not independently established
- Introduction · Benchmark descriptions: CyberSOCEval and AutoPatch TestsSelected original resource https://meta-llama.github.io/PurpleLlama/CyberSecEval/docs/intro; sha256:d1fa0b20e03bb8672ef79d157f38f9167109068cfcf0ad7f12ada2abcc3457f7; archived source capture; exact capture time not independently established
Original source charts and materials
Evaluation suite
Component protocols
This is a suite of distinct, versioned protocols. The original introduction does not provide a unified numerical result; component results must be recorded separately.
- 01MITRE / false-refusal tests
- 02Secure code generation
- 03Prompt injection
- 04Code interpreter
- 05Vulnerability exploitation
- 06Spear phishing
- 07Autonomous cyber operations
- 08AutoPatch
- 09Malware analysis
- 10Threat intelligence reasoning
Related research questions
References
- [1] Introduction ; Official publication masthead and article byline