NUWA 评测档案
Agent Security Bench (ASB)
智能体的哪些接口暴露出不同的攻击与防御失效模式？
跨报告监测
已发表证据
Agent Security Bench (ASB) · Attack success rate
跨来源报告的已发表测量。任务范围通过注释区分，配置差异保留在每条观测中。
234 条观测 · 0 已定位日期
● 条件匹配◇ 配置差异□ 部分条件未披露
- DPI
- IPI
- Memory poisoning
- Mixed attack
- PoT backdoor
- Average across attack types
- DPI baseline
- Delimiter
- Paraphrase
- Instructional
- Dynamic prompt rewriting
- DPI+IPI
- DPI+MP
- IPI+MP
- DPI+IPI+MP
- PoT baseline
- Shuffle defense
- Paraphrase defense
234 条观测尚无已核实的模型发布日期，可在数据与系统结果中查看。
选择图中数据点或表格行，查看条件与出处。
234 条观测
| 系统／模型 | 结果 | 条件状态 | 任务／条件 | 评测方／报告 | 日期 |
|---|---|---|---|---|---|
 Gemma2-27B | 100%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
 GPT-4o | 100%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 100%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 100%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 100%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 99.75%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 99.75%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 99.5%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 99.5%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 99.5%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 99.25%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 99%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 98.75%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
 Qwen2-72B | 98.5%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 98.5%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 98.5%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 98.4%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 98%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 97.75%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 97.5%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 96.75%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 96.75%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 96.75%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 95.75%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
 LLaMA3.1-70B | 95.75%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 95.5%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 95.5%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 95.5%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 95.45%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 95%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 94.5%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude3.5 Sonnet | 94.5%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 94.5%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 94.5%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
 Claude-3.5 Sonnet | 94.5%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 94.5%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 93.5%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 93.5%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 92.75%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 92.17%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 92.17%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 91.75%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 91.75%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 91%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 91%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude3.5 Sonnet | 90.75%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 90.5%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 89.25%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 89.25%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 88.75%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 88.5%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 88.25%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 88.25%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 88%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 88%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 87.75%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 87.75%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 87.25%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 87.1%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 86.95%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 86.5%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 86.15%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 85.65%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 85.5%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 85.5%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 84.75%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 83.75%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 80%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 79%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 78.75%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 78.25%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 76.75%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 76%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 76%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 76%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 74.5%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 73.5%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 73.5%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 73.5%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 71.5%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 71.25%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 70.75%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 70.25%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 70%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 69%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 68.75%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 68.25%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 68%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 67.55%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 67.5%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 67.5%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 67.25%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 65.25%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 65%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 64.41%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 64.25%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 63.75%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 63.5%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 62.75%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 62.75%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 62.5%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 62.5%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 62.45%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 62.25%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 61.25%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 60.5%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 60.35%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 60.2%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 59.75%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 59.75%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude3.5 Sonnet | 59.7%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 59.5%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 58.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 57.75%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 57.75%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 57%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 57%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 56.75%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude3.5 Sonnet | 56.44%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 55.75%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 55.75%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 55.5%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
 Mixtral-8x7B | 55.5%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 55.2%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 55.1%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 54.84%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 54.75%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 54.75%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 54.5%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 54.5%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 54.34%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 54.16%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 53.7%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 53.6%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 52.25%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 52%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 51.1%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 50.97%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 49.5%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 48.01%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 46.25%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 44.75%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 44.6%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 44.55%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 43.7%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 43.25%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 43%Attack success rate | 条件不完整 | DelimiterASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 42.5%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 42%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 41.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 40.75%Attack success rate | 条件不完整 | Mixed attackASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 40.75%Attack success rate | 条件不完整 | DPI+IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 39.75%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 39.75%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 39.75%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 39.4%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 38.5%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 37.75%Attack success rate | 条件不完整 | DPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 37.75%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 35.8%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 35.13%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 34%Attack success rate | 条件不完整 | InstructionalASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 33.75%Attack success rate | 条件不完整 | DPI baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 33.3%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 32.5%Attack success rate | 条件不完整 | DPI+IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 31.06%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 28.5%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 27.5%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 26.7%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 25.85%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 25.65%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 25.2%Attack success rate | 条件不完整 | DPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 24.5%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 23.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 23.25%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 23.15%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 22.75%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 21.5%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 21.5%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 21.35%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 21%Attack success rate | 条件不完整 | ParaphraseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 20.26%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude3.5 Sonnet | 19.75%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 19.01%Attack success rate | 条件不完整 | Average across attack typesASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 19%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 19%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude3.5 Sonnet | 17.5%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 17.5%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 17.5%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 17.25%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 16.5%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 16%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 14.5%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 14.2%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 14.2%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 14%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Claude-3.5 Sonnet | 13.75%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 12.25%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 12.25%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 12.25%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 12.1%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 11%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 10.55%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 10.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o | 10%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 9.3%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 9.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 9%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 8.35%Attack success rate | 条件不完整 | Dynamic prompt rewritingASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 8.25%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 8.25%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-9B | 6.85%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 6.4%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Gemma2-27B | 6.25%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-8B | 6.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-3.5 Turbo | 6.25%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| GPT-4o-mini | 5.5%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 5%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 4.9%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 4.8%Attack success rate | 条件不完整 | IPIASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 4.75%Attack success rate | 条件不完整 | PoT backdoorASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 4.75%Attack success rate | 条件不完整 | PoT baselineASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 4.5%Attack success rate | 条件不完整 | Paraphrase defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 4.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Mixtral-8x7B | 4.25%Attack success rate | 条件不完整 | IPI+MPASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-72B | 3.95%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 3.3%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3.1-70B | 2.85%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| Qwen2-7B | 2.85%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-8B | 2.25%Attack success rate | 条件不完整 | Shuffle defenseASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
| LLaMA3-70B | 1.85%Attack success rate | 条件不完整 | Memory poisoningASB ReAct framework based on AIOS | 未披露Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents | 发布 —报告 2024-10-03 |
结合任务范围与运行配置解读结果。
METR Time Horizon 1.1 来源已停止积极更新，最后更新于 2026 年 9 月 8 日。超过 16 小时的估计超出其可靠测量范围。
协议、配置与原始证据 →任务与测量
Benchmark 协议档案
- 任务／对象
- ASB 区分系统与用户提示、工具响应和检索记忆，在共同智能体框架中评测攻击与防御。
- 测量目标
- 智能体的哪些接口暴露出不同的攻击与防御失效模式？
- 评分／结果口径
- Attack success rate · Attack success rate
- 解释边界
- 攻击面、基础模型和防御配置不同；汇总攻击率不是部署风险概率。方法与指标数量随版本变化，不能沿用旧摘要数字。
一项已报告评测
来源评测 Gemma2-9B · DPI.
运行配置: ASB ReAct framework based on AIOS.
报告指标: Attack success rate · 87.1%.
Attack success rate 见原文样本说明
本次捕获版本报告多个接口存在脆弱性，所测防御保护有限；安全性需要与任务效用共同检查。
测量档案 · NUWA 解读
档案区分来源方法阅读与既有目录信息；已经公开不代表完成独立证据复核或双语编辑复核。
- 测量构念
- 智能体提示、工具响应与记忆中的攻击和防御失效。
- 任务与样本
- 十类场景的智能体执行工具任务；直接／间接提示注入、记忆投毒、混合攻击与思维计划后门分别作为条件。
- 判分与失效情形
- ASR 按被攻击任务中使用攻击工具的比例计算；RR 测量拒绝，PNA 测量正常任务完成。各指标及其聚合分别保留。
- 系统条件
- ASB 采用基于 AIOS 的 ReAct 框架，报告 13 种基础模型。附录 C.2.2 说明基于 Chroma 的记忆检索；每条观测保留具体攻击和工具条件。
- 统计含义
- 现有材料尚未确定
- 效度与解释
- Table 6 重用 Table 5 的 ASR 和 Table 19 的 PNA，不作为新运行。Table 13 的闭源模型参数估计不用于模型尺寸编码。
- 适用研究问题
- 现有材料尚未确定
- Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents · #S3; #S5; #S685bab447cf5c3c692c7d3dae2785f0bcc6a9e4f8d8e15da9aac5cb6ec976054b
跨报告证据
相同 Benchmark 名称可能对应不同任务、计分规则和系统配置。各来源分别保留，比较组说明具体对齐了哪些条件。
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
实际评测方: 未披露
611 条结果记录 · 14 个报告系统名称
协议与比较依据 (0)
原始来源图表与材料
攻击面、基础模型和防御配置不同；汇总攻击率不是部署风险概率。方法与指标数量随版本变化，不能沿用旧摘要数字。
相关研究问题
参考文献
数值结果由所引研究的作者报告。
- [1] Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. 2024-10-03 ; #S5.T5; row Gemma2-9B; condition DPI; column Attack success rate; #S5.T5; row Gemma2-9B; condition DPI; column Refusal rate; #S5.T5; row Gemma2-9B; condition IPI; column Attack success rate