NUWA Evaluation dossier
Agent Security Bench (ASB)
Which agent interfaces expose distinct attack and defense failure modes?
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
Agent Security Bench (ASB) · Attack success rate
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
234 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
- DPI
- IPI
- Memory poisoning
- Mixed attack
- PoT backdoor
- Average across attack types
- DPI baseline
- Delimiter
- Paraphrase
- Instructional
- Dynamic prompt rewriting
- DPI+IPI
- DPI+MP
- IPI+MP
- DPI+IPI+MP
- PoT baseline
- Shuffle defense
- Paraphrase defense
234 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
234 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 Gemma2-27B | 100%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
 GPT-4o | 100%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 100%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 100%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 100%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 99.75%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 99.5%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 99.5%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 99.5%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 99.25%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 99%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 98.75%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
 Qwen2-72B | 98.5%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 98.5%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 98.5%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 98.4%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 98%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 97.75%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 97.5%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 96.75%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 96.75%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 96.75%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 95.75%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
 LLaMA3.1-70B | 95.75%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 95.5%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 95.5%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 95.5%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 95.45%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 95%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 94.5%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude3.5 Sonnet | 94.5%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 94.5%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 94.5%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
 Claude-3.5 Sonnet | 94.5%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 94.5%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 93.5%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 93.5%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 92.75%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 92.17%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 92.17%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 91.75%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 91.75%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 91%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 91%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude3.5 Sonnet | 90.75%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 90.5%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 89.25%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 89.25%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 88.75%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 88.5%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 88.25%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 88.25%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 88%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 88%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 87.75%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 87.75%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 87.25%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 87.1%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 86.95%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 86.5%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 86.15%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 85.65%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 85.5%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 85.5%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 84.75%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 83.75%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 80%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 79%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 78.75%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 78.25%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 76.75%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 76%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 76%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 76%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 74.5%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 73.5%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 73.5%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 73.5%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 71.5%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 71.25%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 70.75%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 70.25%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 70%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 69%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 68.75%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 68.25%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 68%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 67.55%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 67.5%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 67.5%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 67.25%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 65.25%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 65%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 64.41%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 64.25%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 63.75%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 63.5%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 62.75%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 62.75%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 62.5%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 62.5%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 62.45%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 62.25%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 61.25%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 60.5%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 60.35%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 60.2%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 59.75%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 59.75%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude3.5 Sonnet | 59.7%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 59.5%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 58.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 57.75%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 57.75%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 57%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 57%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 56.75%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude3.5 Sonnet | 56.44%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 55.75%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 55.75%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 55.5%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
 Mixtral-8x7B | 55.5%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 55.2%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 55.1%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 54.84%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 54.75%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 54.75%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 54.5%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 54.5%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 54.34%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 54.16%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 53.7%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 53.6%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 52.25%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 52%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 51.1%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 50.97%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 49.5%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 48.01%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 46.25%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 44.75%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 44.6%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 44.55%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 43.7%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 43.25%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 43%Attack success rate | Incomplete | DelimiterASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 42.5%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 42%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 41.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 40.75%Attack success rate | Incomplete | Mixed attackASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 40.75%Attack success rate | Incomplete | DPI+IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 39.75%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 39.75%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 39.75%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 39.4%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 38.5%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 37.75%Attack success rate | Incomplete | DPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 37.75%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 35.8%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 35.13%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 34%Attack success rate | Incomplete | InstructionalASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 33.75%Attack success rate | Incomplete | DPI baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 33.3%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 32.5%Attack success rate | Incomplete | DPI+IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 31.06%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 28.5%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 27.5%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 26.7%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 25.85%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 25.65%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 25.2%Attack success rate | Incomplete | DPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 24.5%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 23.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 23.25%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 23.15%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 22.75%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 21.5%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 21.5%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 21.35%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 21%Attack success rate | Incomplete | ParaphraseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 20.26%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude3.5 Sonnet | 19.75%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 19.01%Attack success rate | Incomplete | Average across attack typesASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 19%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 19%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude3.5 Sonnet | 17.5%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 17.5%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 17.5%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 17.25%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 16.5%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 16%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 14.5%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 14.2%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 14.2%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 14%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Claude-3.5 Sonnet | 13.75%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 12.25%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 12.25%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 12.25%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 12.1%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 11%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 10.55%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 10.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o | 10%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 9.3%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 9.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 9%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 8.35%Attack success rate | Incomplete | Dynamic prompt rewritingASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 8.25%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 8.25%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-9B | 6.85%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 6.4%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Gemma2-27B | 6.25%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-8B | 6.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-3.5 Turbo | 6.25%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| GPT-4o-mini | 5.5%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 5%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 4.9%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 4.8%Attack success rate | Incomplete | IPIASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 4.75%Attack success rate | Incomplete | PoT backdoorASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 4.75%Attack success rate | Incomplete | PoT baselineASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 4.5%Attack success rate | Incomplete | Paraphrase defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 4.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Mixtral-8x7B | 4.25%Attack success rate | Incomplete | IPI+MPASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-72B | 3.95%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 3.3%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3.1-70B | 2.85%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| Qwen2-7B | 2.85%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-8B | 2.25%Attack success rate | Incomplete | Shuffle defenseASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
| LLaMA3-70B | 1.85%Attack success rate | Incomplete | Memory poisoningASB ReAct framework based on AIOS | Not disclosedAgent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | Release —Report 2024-10-03 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- ASB separates system/user prompts, tool responses and retrieved memory, then evaluates attacks and defenses on a shared agent framework.
- Measurement target
- Which agent interfaces expose distinct attack and defense failure modes?
- Scoring / outcome
- Attack success rate · Attack success rate
- Interpretation boundary
- Attack surfaces, backbones, and defense configurations differ; its aggregate attack rate is not a deployment probability. Revision-dependent method and metric counts differ from older abstracts.
A reported evaluation
The source evaluates Gemma2-9B · DPI.
Run configuration: ASB ReAct framework based on AIOS.
Reported measurement: Attack success rate · 87.1%.
Attack success rate Not disclosed
The captured revision reports vulnerability across multiple interfaces and limited protection from the evaluated defenses. Utility and security must be assessed together.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Attack and defense failure modes across agent prompts, tool responses and memory.
- Task population
- Ten scenario-specific agents carry out tool-using tasks. Direct/indirect prompt injection, memory poisoning, mixed attacks and plan-of-thought backdoors are separate conditions.
- Scoring and failure modes
- ASR counts attacked tasks in which attack-specific tools are used; RR measures refusal. PNA measures normal task completion. These metrics and their aggregates remain separate.
- System conditions
- ASB uses a ReAct framework based on AIOS. The paper reports 13 LLM backbones. Appendix C.2.2 describes memory retrieval through Chroma; per-condition tool/attack settings are retained with each observation.
- Statistical meaning
- Not established by the retained material
- Validity and interpretation
- Table 6 reuses ASR from Table 5 and PNA from Table 19, rather than new runs. Proprietary-model parameter estimates in Table 13 are not adopted as model-size evidence.
- Appropriate research use
- Not established by the retained material
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents · #S3; #S5; #S685bab447cf5c3c692c7d3dae2785f0bcc6a9e4f8d8e15da9aac5cb6ec976054b
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Actual evaluators: Not disclosed
611 result records · 14 reported system names
Protocols and comparison decisions (0)
Original source charts and materials
Attack surfaces, backbones, and defense configurations differ; its aggregate attack rate is not a deployment probability. Revision-dependent method and metric counts differ from older abstracts.
Related research questions
References
Numerical results are reported by the cited authors.
- [1] Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. 2024-10-03 ; #S5.T5; row Gemma2-9B; condition DPI; column Attack success rate; #S5.T5; row Gemma2-9B; condition DPI; column Refusal rate; #S5.T5; row Gemma2-9B; condition IPI; column Attack success rate