NUWA Evidence record
GPT-3.5 / ReAct
RedCode · Evidence record
Original source charts and materials
Recorded measurements
- Attack Success Rate
- 51.47%
Not disclosed
ReAct · temperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)
Method and scope
Does a code agent recognize unsafe execution and unsafe generation under different input forms?
- Evidence type
- Benchmark
- Assessment subject
- RedCode separates execution from generation; execution runs in containers with outcome checks, while generation uses different quality assessments.
- Primary dimension
- Behavior
- Evidence snapshot
- 2026-10-29.1
Execution and generation scores have different judges and denominators. Agent frameworks support different language subsets; comparing their aggregate rates without matching those subsets is inappropriate.
Source finding
The reported behavior depends on whether a risky request is expressed as code or prose and on the type of unsafe behavior. Coding capability alone does not establish safe execution.
Limitations
Execution and generation scores have different judges and denominators. Agent frameworks support different language subsets; comparing their aggregate rates without matching those subsets is inappropriate.
Execution and generation scores have different judges and denominators. Agent frameworks support different language subsets; comparing their aggregate rates without matching those subsets is inappropriate.
Related research questions
References
Numerical results are reported by the cited authors.
- [1] RedCode: Risky Code Execution and Generation Benchmark for Code Agents. 2024-11-12 ; #A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (b) Awareness; column Attack Success Rate