NUWA Evaluation dossier
AgentDojo
Agent behavior when untrusted tool outputs contain adversarial instructions.
CROSS-REPORT MONITORING
PUBLISHED EVIDENCE
AgentDojo · Utility under attack
Published measurements across source reports. Task scopes are annotated; configuration differences remain attached to each observation.
10 observations · 0 dated
● Matched◇ Configuration differs□ Some conditions undisclosed
10 observations lack a verified model release date. They remain in Data and Systems.
Select a point or table row to inspect its conditions and provenance.
10 observations
| System / model | Result | Condition status | Task / conditions | Evaluator / report | Dates |
|---|---|---|---|---|---|
 GPT-4 Turbo | 54.05%Utility under attack± 3.89 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
 Claude 3 Opus | 52.46%Utility under attack± 3.90 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
| Claude 3.5 Sonnet | 51.19%Utility under attack± 3.91 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
| GPT-4o | 50.08%Utility under attack± 3.91 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
| GPT-3.5 Turbo | 34.66%Utility under attack± 3.72 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
 Gemini 1.5 Flash | 34.18%Utility under attack± 3.71 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
| Claude 3 Sonnet | 33.23%Utility under attack± 3.68 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
| Gemini 1.5 Pro | 28.93%Utility under attack± 3.54 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
 Command-R+ | 25.12%Utility under attack± 3.39 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
 Llama 3 70b | 18.28%Utility under attack± 3.02 pp (95% CI) | Incomplete | 629 security cases / 97 user tasksPaper's baseline tool-calling agent (§3.2) | Not disclosedAgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents | Release —Report 2024-11-24 |
Compare results within the stated task and configuration.
METR Time Horizon 1.1: the source is no longer actively updated (last update: 8 September 2026). Estimates above 16 hours exceed its reliable measurement range.
Protocol, configurations and original evidence →Task and measurement
Benchmark protocol profile
- Task / subject
- Defender agent under injected tool output
- Measurement target
- Targeted attack success and user-task utility in a tool-use environment.
- Scoring / outcome
- Benign utility · User-task utility without attack.
- Interpretation boundary
- Attack success must be read alongside benign utility and the tested attack setup.
A reported evaluation
The source evaluates Claude 3 Opus · 629 security cases / 97 user tasks.
Run configuration: Paper's baseline tool-calling agent (§3.2).
Reported measurement: Benign utility · 66.61%.
User-task utility without attack. 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
The source reports these three outcomes for the named baseline agent.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can tool-using agents maintain utility while processing attacker-controlled external data?
- Task population
- An executable, stateful environment includes 97 user tasks and 629 security cases, pairing a user task with an adversarial objective.
- Scoring and failure modes
- Benign utility, targeted attack success and legitimate utility without adversarial side effects are distinct.
- System conditions
- 97 user tasks, 629 paired security cases in executable stateful applications.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: Utility under attack requires completing the legitimate task without adversarial side effects (§3.4), rather than merely refusing an injection.
- Validity and interpretation
- Tool isolation fails when tools cannot be planned in advance or legitimate tools suffice for the attack; ongoing multi-task contexts are incompletely covered.
- Appropriate research use
- Model recommendation: report benign utility, utility under attack and targeted attack success separately.
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · AbstractSelected original resource https://arxiv.org/html/2406.13352v3; sha256:7b6e97faaedcf0e1b5f3635b92a0da33afbe8529699d54a5c44ad58ce2fa8b02; archived source capture; exact capture time not independently established
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · §3 Designing and Constructing AgentDojoSelected original resource https://arxiv.org/html/2406.13352v3; sha256:7b6e97faaedcf0e1b5f3635b92a0da33afbe8529699d54a5c44ad58ce2fa8b02; archived source capture; exact capture time not independently established
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · §3.4 Reporting AgentDojo ResultsSelected original resource https://arxiv.org/html/2406.13352v3; sha256:7b6e97faaedcf0e1b5f3635b92a0da33afbe8529699d54a5c44ad58ce2fa8b02; archived source capture; exact capture time not independently established
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents · §4.3 Prompt Injection DefensesSelected original resource https://arxiv.org/html/2406.13352v3; sha256:7b6e97faaedcf0e1b5f3635b92a0da33afbe8529699d54a5c44ad58ce2fa8b02; archived source capture; exact capture time not independently established
Evidence across reports
The same benchmark name can cover different tasks, scoring rules and system configurations. Each source remains visible; comparison groups explain which conditions are aligned.
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents
Actual evaluators: Not disclosed
10 result records · 10 reported system names
Protocols and comparison decisions (3)
AgentDojo: Benign utility · AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentssource-report · Scope reviewed
NIST / CAISI
Evaluation of DeepSeek AI Models →
Actual evaluators: NIST / CAISI · Not disclosed
9 result records · 3 reported system names
Protocols and comparison decisions (3)
AgentDojo: Hijacking attempted-task frequency · NIST / CAISIsource-report · Scope reviewed
AgentDojo: Hijacking attempted-task frequency · NIST / CAISIsource-report · Scope reviewed
AgentDojo: Hijacking attempted-task frequency · NIST / CAISIsource-report · Scope reviewed
Original source charts and materials
Reported results
Source order · reported configurations
Select systems
Targeted attack success
0–100%Claude 3 OpusClaude 3 Opus · 11.29% · ± 2.47 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
11.29%Claude 3 SonnetClaude 3 Sonnet · 26.71% · ± 3.46 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
26.71%Claude 3.5 SonnetClaude 3.5 Sonnet · 33.86% · ± 3.70 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
33.86%Command-R+Command-R+ · 0.95% · ± 0.76 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
0.95%Gemini 1.5 FlashGemini 1.5 Flash · 12.24% · ± 2.56 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
12.24%Gemini 1.5 ProGemini 1.5 Pro · 25.60% · ± 3.41 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
25.60%GPT-3.5 TurboGPT-3.5 Turbo · 8.43% · ± 2.17 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
8.43%GPT-4 TurboGPT-4 Turbo · 28.62% · ± 3.53 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
28.62%GPT-4oGPT-4o · 47.69% · ± 3.90 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
47.69%Llama 3 70bLlama 3 70b · 20.03% · ± 3.13 pp (95% CI) · Attacker-objective completion, not a safety score. · Paper's baseline tool-calling agent (§3.2) · 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions.
20.03%Showing 10 of 10 systems
Values describe the measured outcome under the source protocol, not a general safety score.
Table 3. Baseline agents on 629 security cases covering 97 user tasks. Error bars are the source-reported 95% confidence intervals; cases are not 629 independent trials. Evaluation date not specified; publication / record dates are shown below.
Evaluation conditions and system configurations
- Claude 3 Opus
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- Claude 3 Sonnet
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- Claude 3.5 Sonnet
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- Command-R+
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- Gemini 1.5 Flash
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- Gemini 1.5 Pro
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- GPT-3.5 Turbo
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- GPT-4 Turbo
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- GPT-4o
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
- Llama 3 70b
- 629 security test cases; 97 distinct user tasks. Not 629 independent repetitions. · 629 security cases / 97 user tasks
- modelVersion: not specified · harness: Paper's baseline tool-calling agent (§3.2) · tools: Workspace, Slack, banking and travel task suites · budget: not specified
| System / configuration | Benign utility | Utility under attack | Targeted attack success | Publication / verification |
|---|---|---|---|---|
| Claude 3 OpusPaper's baseline tool-calling agent (§3.2) | 66.61%± 3.69 pp (95% CI) | 52.46%± 3.90 pp (95% CI) | 11.29%± 2.47 pp (95% CI) | Published2024-11-24 |
| Claude 3 SonnetPaper's baseline tool-calling agent (§3.2) | 53.10%± 3.90 pp (95% CI) | 33.23%± 3.68 pp (95% CI) | 26.71%± 3.46 pp (95% CI) | Published2024-11-24 |
| Claude 3.5 SonnetPaper's baseline tool-calling agent (§3.2) | 78.22%± 3.23 pp (95% CI) | 51.19%± 3.91 pp (95% CI) | 33.86%± 3.70 pp (95% CI) | Published2024-11-24 |
| Command-R+Paper's baseline tool-calling agent (§3.2) | 25.44%± 3.40 pp (95% CI) | 25.12%± 3.39 pp (95% CI) | 0.95%± 0.76 pp (95% CI) | Published2024-11-24 |
| Gemini 1.5 FlashPaper's baseline tool-calling agent (§3.2) | 36.09%± 3.75 pp (95% CI) | 34.18%± 3.71 pp (95% CI) | 12.24%± 2.56 pp (95% CI) | Published2024-11-24 |
| Gemini 1.5 ProPaper's baseline tool-calling agent (§3.2) | 45.63%± 3.89 pp (95% CI) | 28.93%± 3.54 pp (95% CI) | 25.60%± 3.41 pp (95% CI) | Published2024-11-24 |
| GPT-3.5 TurboPaper's baseline tool-calling agent (§3.2) | 33.86%± 3.70 pp (95% CI) | 34.66%± 3.72 pp (95% CI) | 8.43%± 2.17 pp (95% CI) | Published2024-11-24 |
| GPT-4 TurboPaper's baseline tool-calling agent (§3.2) | 63.43%± 3.76 pp (95% CI) | 54.05%± 3.89 pp (95% CI) | 28.62%± 3.53 pp (95% CI) | Published2024-11-24 |
| GPT-4oPaper's baseline tool-calling agent (§3.2) | 69.00%± 3.61 pp (95% CI) | 50.08%± 3.91 pp (95% CI) | 47.69%± 3.90 pp (95% CI) | Published2024-11-24 |
| Llama 3 70bPaper's baseline tool-calling agent (§3.2) | 34.50%± 3.71 pp (95% CI) | 18.28%± 3.02 pp (95% CI) | 20.03%± 3.13 pp (95% CI) | Published2024-11-24 |
Assessment provenance
Evaluation of DeepSeek AI Models → · External report
Related research questions
Cybersecurity / Agent compromise through untrusted input →
The source directly measures the stated construct under its protocol.
References
- [1] deepseek-r1-0528 release ; News article title, dated heading and availability
- [2] OpenAI — introducing gpt oss ; Publication date and availability section
- [3] gpt-oss-120b developer model card ; Introduction: gpt-oss-120b parameters
- [4] Edoardo Debenedetti, Jie Zhang, et al.. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. 2024-11-24. v3 ; Author affiliations on the original paper title page; Appendix C, Table 3
- [5] Evaluation of DeepSeek AI Models. 2025-09-30. 30 September 2025 ; p. 17, key findings, Credential exfiltration; p. 17, key findings, Phishing email; p. 17, key findings, Download/run malware