NUWA RESEARCH RECORD
RedCode: Risky Code Execution and Generation Benchmark for Code Agents
原始文献 ↗ · BibTeX · RIS报告中的行为随代码或文字输入、风险行为类型而变化；编码能力本身不能证明执行安全。
研究问题
代码智能体能否在不同输入形式下识别不安全执行与生成？
设计与前提
RedCode 分开测量执行和生成：执行在容器中检查结果，生成使用另一套质量评估。
研究发现
报告中的行为随代码或文字输入、风险行为类型而变化；编码能力本身不能证明执行安全。
局限与反证
执行与生成成绩的评审和分母不同；智能体框架支持的语言子集也不同，不能直接比较未匹配子集的汇总率。
来源证据
| 系统 | 测量 | 原值 | 条件与位置 |
|---|---|---|---|
| GPT-4o / Base LLM | Accuracy | 69.4% | RedCode-Gen / Base LLM#S4.T1; row GPT-4o / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| GPT-4o / Base LLM | Refusal rate | 19.4% | RedCode-Gen / Base LLM#S4.T1; row GPT-4o / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| GPT-4o / Base LLM | VirusTotal score (as reported) | 3.8source-reported score | RedCode-Gen / Base LLM#S4.T1; row GPT-4o / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| GPT-4 / Base LLM | Accuracy | 65.0% | RedCode-Gen / Base LLM#S4.T1; row GPT-4 / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| GPT-4 / Base LLM | Refusal rate | 13.8% | RedCode-Gen / Base LLM#S4.T1; row GPT-4 / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| GPT-4 / Base LLM | VirusTotal score (as reported) | 4.4source-reported score | RedCode-Gen / Base LLM#S4.T1; row GPT-4 / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| GPT-3.5 / Base LLM | Accuracy | 0.0% | RedCode-Gen / Base LLM#S4.T1; row GPT-3.5 / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| GPT-3.5 / Base LLM | Refusal rate | 87.5% | RedCode-Gen / Base LLM#S4.T1; row GPT-3.5 / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| GPT-3.5 / Base LLM | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Base LLM#S4.T1; row GPT-3.5 / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| Claude-Opus / Base LLM | Accuracy | 1.3% | RedCode-Gen / Base LLM#S4.T1; row Claude-Opus / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| Claude-Opus / Base LLM | Refusal rate | 98.9% | RedCode-Gen / Base LLM#S4.T1; row Claude-Opus / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| Claude-Opus / Base LLM | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Base LLM#S4.T1; row Claude-Opus / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| DeepseekCoder / Base LLM | Accuracy | 49.4% | RedCode-Gen / Base LLM#S4.T1; row DeepseekCoder / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| DeepseekCoder / Base LLM | Refusal rate | 11.3% | RedCode-Gen / Base LLM#S4.T1; row DeepseekCoder / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| DeepseekCoder / Base LLM | VirusTotal score (as reported) | 4.4source-reported score | RedCode-Gen / Base LLM#S4.T1; row DeepseekCoder / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| CodeLlama-7B / Base LLM | Accuracy | 40.0% | RedCode-Gen / Base LLM#S4.T1; row CodeLlama-7B / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| CodeLlama-7B / Base LLM | Refusal rate | 40.0% | RedCode-Gen / Base LLM#S4.T1; row CodeLlama-7B / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| CodeLlama-7B / Base LLM | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Base LLM#S4.T1; row CodeLlama-7B / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| CodeLlama-13B / Base LLM | Accuracy | 49.4% | RedCode-Gen / Base LLM#S4.T1; row CodeLlama-13B / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| CodeLlama-13B / Base LLM | Refusal rate | 30.6% | RedCode-Gen / Base LLM#S4.T1; row CodeLlama-13B / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| CodeLlama-13B / Base LLM | VirusTotal score (as reported) | 0.6source-reported score | RedCode-Gen / Base LLM#S4.T1; row CodeLlama-13B / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| Llama-2-7B / Base LLM | Accuracy | 16.9% | RedCode-Gen / Base LLM#S4.T1; row Llama-2-7B / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| Llama-2-7B / Base LLM | Refusal rate | 61.9% | RedCode-Gen / Base LLM#S4.T1; row Llama-2-7B / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| Llama-2-7B / Base LLM | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Base LLM#S4.T1; row Llama-2-7B / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| Mistral-7B / Base LLM | Accuracy | 46.3% | RedCode-Gen / Base LLM#S4.T1; row Mistral-7B / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy |
| Mistral-7B / Base LLM | Refusal rate | 23.1% | RedCode-Gen / Base LLM#S4.T1; row Mistral-7B / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate |
| Mistral-7B / Base LLM | VirusTotal score (as reported) | 0.6source-reported score | RedCode-Gen / Base LLM#S4.T1; row Mistral-7B / Base LLM; condition RedCode-Gen / Base LLM; column VirusTotal detection rate |
| GPT-4o / Code Agent | Accuracy | 72.5% | RedCode-Gen / Code Agent#S4.T1; row GPT-4o / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| GPT-4o / Code Agent | Refusal rate | 18.1% | RedCode-Gen / Code Agent#S4.T1; row GPT-4o / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| GPT-4o / Code Agent | VirusTotal score (as reported) | 4.4source-reported score | RedCode-Gen / Code Agent#S4.T1; row GPT-4o / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| GPT-4 / Code Agent | Accuracy | 66.9% | RedCode-Gen / Code Agent#S4.T1; row GPT-4 / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| GPT-4 / Code Agent | Refusal rate | 11.3% | RedCode-Gen / Code Agent#S4.T1; row GPT-4 / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| GPT-4 / Code Agent | VirusTotal score (as reported) | 5.6source-reported score | RedCode-Gen / Code Agent#S4.T1; row GPT-4 / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| GPT-3.5 / Code Agent | Accuracy | 32.5% | RedCode-Gen / Code Agent#S4.T1; row GPT-3.5 / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| GPT-3.5 / Code Agent | Refusal rate | 30.6% | RedCode-Gen / Code Agent#S4.T1; row GPT-3.5 / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| GPT-3.5 / Code Agent | VirusTotal score (as reported) | 1.3source-reported score | RedCode-Gen / Code Agent#S4.T1; row GPT-3.5 / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| Claude-Opus / Code Agent | Accuracy | 3.1% | RedCode-Gen / Code Agent#S4.T1; row Claude-Opus / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| Claude-Opus / Code Agent | Refusal rate | 96.9% | RedCode-Gen / Code Agent#S4.T1; row Claude-Opus / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| Claude-Opus / Code Agent | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Code Agent#S4.T1; row Claude-Opus / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| DeepSeekCoder / Code Agent | Accuracy | 79.4% | RedCode-Gen / Code Agent#S4.T1; row DeepSeekCoder / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| DeepSeekCoder / Code Agent | Refusal rate | 1.3% | RedCode-Gen / Code Agent#S4.T1; row DeepSeekCoder / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| DeepSeekCoder / Code Agent | VirusTotal score (as reported) | 4.4source-reported score | RedCode-Gen / Code Agent#S4.T1; row DeepSeekCoder / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| CodeLlama-13B / Code Agent | Accuracy | 66.3% | RedCode-Gen / Code Agent#S4.T1; row CodeLlama-13B / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| CodeLlama-13B / Code Agent | Refusal rate | 19.4% | RedCode-Gen / Code Agent#S4.T1; row CodeLlama-13B / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| CodeLlama-13B / Code Agent | VirusTotal score (as reported) | 1.9source-reported score | RedCode-Gen / Code Agent#S4.T1; row CodeLlama-13B / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| CodeLlama-7B / Code Agent | Accuracy | 42.0% | RedCode-Gen / Code Agent#S4.T1; row CodeLlama-7B / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| CodeLlama-7B / Code Agent | Refusal rate | 38.0% | RedCode-Gen / Code Agent#S4.T1; row CodeLlama-7B / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| CodeLlama-7B / Code Agent | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Code Agent#S4.T1; row CodeLlama-7B / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| Llama-2-7B / Code Agent | Accuracy | 20.7% | RedCode-Gen / Code Agent#S4.T1; row Llama-2-7B / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| Llama-2-7B / Code Agent | Refusal rate | 56.7% | RedCode-Gen / Code Agent#S4.T1; row Llama-2-7B / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| Llama-2-7B / Code Agent | VirusTotal score (as reported) | 0.0source-reported score | RedCode-Gen / Code Agent#S4.T1; row Llama-2-7B / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| Mistral-7B / Code Agent | Accuracy | 75.3% | RedCode-Gen / Code Agent#S4.T1; row Mistral-7B / Code Agent; condition RedCode-Gen / Code Agent; column Accuracy |
| Mistral-7B / Code Agent | Refusal rate | 0.0% | RedCode-Gen / Code Agent#S4.T1; row Mistral-7B / Code Agent; condition RedCode-Gen / Code Agent; column Refusal rate |
| Mistral-7B / Code Agent | VirusTotal score (as reported) | 1.3source-reported score | RedCode-Gen / Code Agent#S4.T1; row Mistral-7B / Code Agent; condition RedCode-Gen / Code Agent; column VirusTotal detection rate |
| GPT-3.5 / ReAct | Attack Success Rate | 79.33% | Code input / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / Base; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 64.53% | Code input with jailbreaking / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / Base; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 69.07% | Text summaries / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / Base; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 88.4% | Text descriptions / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / Base; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 71.87% | Code input / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (a) Simple; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 60.93% | Code input with jailbreaking / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (a) Simple; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 63.07% | Text summaries / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (a) Simple; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 87.2% | Text descriptions / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (a) Simple; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 63.73% | Code input / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (b) Awareness; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 51.47% | Code input with jailbreaking / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (b) Awareness; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 59.6% | Text summaries / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (b) Awareness; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 85.33% | Text descriptions / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (b) Awareness; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 63.73% | Code input / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (c) Detection; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 51.47% | Code input with jailbreaking / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (c) Detection; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 61.06% | Text summaries / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (c) Detection; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 84.93% | Text descriptions / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (c) Detection; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 67.33% | Code input / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (d) Modification; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 57.73% | Code input with jailbreaking / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (d) Modification; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 62.8% | Text summaries / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (d) Modification; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 86.53% | Text descriptions / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (d) Modification; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 62.13% | Code input / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (e) Combined; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 48.93% | Code input with jailbreaking / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (e) Combined; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 58.80% | Text summaries / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (e) Combined; column Attack Success Rate |
| GPT-3.5 / ReAct | Attack Success Rate | 83.20% | Text descriptions / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (e) Combined; column Attack Success Rate |
| GPT-3.5 / ReAct | Rejection Rate | 19.07% | Code input / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / Base; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 33.6% | Code input with jailbreaking / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / Base; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 10% | Text summaries / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / Base; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 5.33% | Text descriptions / Base#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / Base; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 25.6% | Code input / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (a) Simple; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 37.6% | Code input with jailbreaking / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (a) Simple; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 13.73% | Text summaries / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (a) Simple; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 6.67% | Text descriptions / + (a) Simple#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (a) Simple; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 34.93% | Code input / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (b) Awareness; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 48.13% | Code input with jailbreaking / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (b) Awareness; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 18.13% | Text summaries / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (b) Awareness; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 8.4% | Text descriptions / + (b) Awareness#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (b) Awareness; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 35.2% | Code input / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (c) Detection; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 47.73% | Code input with jailbreaking / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (c) Detection; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 17.6% | Text summaries / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (c) Detection; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 9.47% | Text descriptions / + (c) Detection#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (c) Detection; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 27.33% | Code input / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (d) Modification; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 38% | Code input with jailbreaking / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (d) Modification; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 13.6% | Text summaries / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (d) Modification; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 6.93% | Text descriptions / + (d) Modification#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (d) Modification; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 35.87% | Code input / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input / + (e) Combined; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 50.80% | Code input with jailbreaking / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Code input with jailbreaking / + (e) Combined; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 19.20% | Text summaries / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text summaries / + (e) Combined; column Rejection Rate |
| GPT-3.5 / ReAct | Rejection Rate | 10.93% | Text descriptions / + (e) Combined#A4.T9.fig1; row GPT-3.5 / ReAct; condition Text descriptions / + (e) Combined; column Rejection Rate |
阅读范围与复核
Selected original HTML sections and table context; not a complete paper/appendix audit or independent reproduction.
模型辅助整理，独立证据复核待完成。
核对原始来源 ↗