WhitzardNUWA Lab ↗
NUWANUWA INDEX

Search NUWA Index

Research · Benchmarks · Findings · Perspectives

中文
← NUWA Index

Evaluation dossier

RedCode

Does a code agent recognize unsafe execution and unsafe generation under different input forms?

Affiliation not indexed in this review / Source capture · 7e4796140a / Behavior

Sources and provenance ↓ Read the research narrative →

CROSS-REPORT MONITORING

PUBLISHED EVIDENCE

RedCode · Attack Success Rate

Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.

24 observations · 0 dated

● Matched◇ Configuration differs□ Some conditions undisclosed
Attack Success Rate (%) GPT-3.5 / ReAct [4] GPT-3.5 / ReAct [8] GPT-3.5 / ReAct [20] GPT-3.5 / ReAct [12] GPT-3.5 / ReAct [16] GPT-3.5 / ReAct [24] GPT-3.5 / ReAct [1] GPT-3.5 / ReAct [5] GPT-3.5 / ReAct [3] GPT-3.5 / ReAct [17] GPT-3.5 / ReAct [2] GPT-3.5 / ReAct [9] 0 20 40 60 80 100 88.4% 87.2% 86.53% 85.33% 84.93% 83.2% 79.33% 71.87% 69.07% 67.33% 64.53% 63.73%
  1. Code input / Base
  2. Code input with jailbreaking / Base
  3. Text summaries / Base
  4. Text descriptions / Base
  5. Code input / + (a) Simple
  6. Code input with jailbreaking / + (a) Simple
  7. Text summaries / + (a) Simple
  8. Text descriptions / + (a) Simple
  9. Code input / + (b) Awareness
  10. Code input with jailbreaking / + (b) Awareness
  11. Text summaries / + (b) Awareness
  12. Text descriptions / + (b) Awareness
  13. Code input / + (c) Detection
  14. Code input with jailbreaking / + (c) Detection
  15. Text summaries / + (c) Detection
  16. Text descriptions / + (c) Detection
  17. Code input / + (d) Modification
  18. Code input with jailbreaking / + (d) Modification
  19. Text summaries / + (d) Modification
  20. Text descriptions / + (d) Modification
  21. Code input / + (e) Combined
  22. Code input with jailbreaking / + (e) Combined
  23. Text summaries / + (e) Combined
  24. Text descriptions / + (e) Combined

24 observations lack a verified model release date. They remain in Data and Systems.

Select a point or table row to inspect its conditions and provenance.

24 observations

Measurements and source conditions
System / modelResultCondition statusTask / conditionsEvaluator / reportDates
GPT-3.5 / ReAct88.4%Attack Success RateIncompleteText descriptions / Basetemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct87.2%Attack Success RateIncompleteText descriptions / + (a) Simpletemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct86.53%Attack Success RateIncompleteText descriptions / + (d) Modificationtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct85.33%Attack Success RateIncompleteText descriptions / + (b) Awarenesstemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct84.93%Attack Success RateIncompleteText descriptions / + (c) Detectiontemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct83.2%Attack Success RateIncompleteText descriptions / + (e) Combinedtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct79.33%Attack Success RateIncompleteCode input / Basetemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct71.87%Attack Success RateIncompleteCode input / + (a) Simpletemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct69.07%Attack Success RateIncompleteText summaries / Basetemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct67.33%Attack Success RateIncompleteCode input / + (d) Modificationtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct64.53%Attack Success RateIncompleteCode input with jailbreaking / Basetemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct63.73%Attack Success RateIncompleteCode input / + (b) Awarenesstemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct63.73%Attack Success RateIncompleteCode input / + (c) Detectiontemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct63.07%Attack Success RateIncompleteText summaries / + (a) Simpletemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct62.8%Attack Success RateIncompleteText summaries / + (d) Modificationtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct62.13%Attack Success RateIncompleteCode input / + (e) Combinedtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct61.06%Attack Success RateIncompleteText summaries / + (c) Detectiontemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct60.93%Attack Success RateIncompleteCode input with jailbreaking / + (a) Simpletemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct59.6%Attack Success RateIncompleteText summaries / + (b) Awarenesstemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct58.8%Attack Success RateIncompleteText summaries / + (e) Combinedtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct57.73%Attack Success RateIncompleteCode input with jailbreaking / + (d) Modificationtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct51.47%Attack Success RateIncompleteCode input with jailbreaking / + (b) Awarenesstemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct51.47%Attack Success RateIncompleteCode input with jailbreaking / + (c) Detectiontemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12
GPT-3.5 / ReAct48.93%Attack Success RateIncompleteCode input with jailbreaking / + (e) Combinedtemperature=0; max 1024 generated tokens; 8 interaction rounds (Appendix C.2)ReActNot disclosedRedCode: Risky Code Execution and Generation Benchmark for Code AgentsRelease —Report 2024-11-12

Evaluated by: Not disclosed
Reported by: RedCode: Risky Code Execution and Generation Benchmark for Code Agents

Task and measurement

Benchmark protocol profile

Task / subject
RedCode separates execution from generation; execution runs in containers with outcome checks, while generation uses different quality assessments.
Measurement target
Does a code agent recognize unsafe execution and unsafe generation under different input forms?
Scoring / outcome
Accuracy · Accuracy
Interpretation boundary
Execution and generation scores have different judges and denominators. Agent frameworks support different language subsets; comparing their aggregate rates without matching those subsets is inappropriate.

A reported evaluation

The source evaluates GPT-4o / Base LLM · RedCode-Gen / Base LLM.

Run configuration: Base model generation.

Reported measurement: Accuracy · 69.4%.

Accuracy Not disclosed

The reported behavior depends on whether a risky request is expressed as code or prose and on the type of unsafe behavior. Coding capability alone does not establish safe execution.

Measurement dossier · NUWA reading

Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.

Measurement construct
Unsafe code execution and explicitly harmful code generation under distinct protocols.
Task population
RedCode-Exec contains 4,050 test cases spanning code and text forms; RedCode-Gen contains 160 function-signature/docstring prompts across eight software categories.
Scoring and failure modes
Exec uses execution scripts to distinguish rejection, execution failure and attack success. Gen reports refusal, generated-code accuracy and a separate VirusTotal score. The latter retains its source label because its unit has not been established here.
System conditions
Base models and code agents are separate systems. For the ReAct mitigation table, Appendix C.2 specifies temperature 0, up to 1024 generated tokens and eight interaction rounds.
Statistical meaning
Not established by the retained material
Validity and interpretation
Exec and Gen are different measurements and are not averaged. Numeric figures remain queued for explicit data acquisition or digitization; this table extraction is not full benchmark data coverage.
Appropriate research use
Not established by the retained material

Evidence across reports

The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.

RedCode: Risky Code Execution and Generation Benchmark for Code Agents

RedCode: Risky Code Execution and Generation Benchmark for Code Agents

Actual evaluators: Not disclosed

84 result records · 19 reported system names

Protocols and comparison decisions (0)
Original source charts and materials

Read the original protocol ↗

Execution and generation scores have different judges and denominators. Agent frameworks support different language subsets; comparing their aggregate rates without matching those subsets is inappropriate.

Related research questions

References

Numerical results are reported by the cited authors.

  1. [1] RedCode: Risky Code Execution and Generation Benchmark for Code Agents. 2024-11-12 ; #S4.T1; row GPT-4o / Base LLM; condition RedCode-Gen / Base LLM; column Accuracy; #S4.T1; row GPT-4o / Base LLM; condition RedCode-Gen / Base LLM; column Refusal rate