NUWA Evaluation dossier
MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers
Studies tool-poisoning attacks against agents using MCP servers and evaluates instruction following under adversarial tool metadata.
Task and measurement
Benchmark protocol profile
- Task / subject
- Model or agent system in the original evaluation configuration
- Measurement target
- Studies tool-poisoning attacks against agents using MCP servers and evaluates instruction following under adversarial tool metadata.
- Interpretation boundary
- Attack effects depend on tool descriptions and the agent configuration; they are not universal MCP compromise rates.
MCPTox pairs benign queries with poisoned tool metadata in MCP-server environments. Test cases are generated with model assistance and manual refinement, then evaluated through the agent’s resulting tool calls.
Measurement dossier · NUWA reading
Source-method reading and retained catalogue information are distinguished. Publication does not establish independent evidence or bilingual editorial review.
- Measurement construct
- Can poisoned tool metadata redirect agents even without invoking the poisoned tool?
- Task population
- 1,348 test cases derive from 45 live MCP servers and 353 tools, using three human-defined attack paradigms and twenty agent settings.
- Scoring and failure modes
- Stealth-specific attack success; calling the poisoned tool directly is a failure.
- System conditions
- 1,348 cases from 45 MCP servers/353 tools; three template paradigms.
- Statistical meaning
- Source-reported result statements; this does not establish trial counts, denominators or an estimator: o1-mini’s reported attack success was 72.8%. Direct invocation of the poisoned tool is labeled failure in the stealth-specific scoring, not success (§4.1).
- Validity and interpretation
- Human-defined generic templates are not adaptive defense-specific attacks; scoring depends on the benchmark’s stealth criterion.
- Appropriate research use
- Model recommendation: preserve metadata exposure and success definition when comparing with AgentDojo or InjecAgent.
- MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers — AAAI-26 paper · AbstractSelected original resource https://arxiv.org/html/2508.14925; sha256:e7326aefa910c4da3b99e93cb2f3defdd93d48c927bf7a89ca3d9c54ec1f89c0; archived source capture; exact capture time not independently established
- MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers — AAAI-26 paper · §2.3 Related BenchmarksSelected original resource https://arxiv.org/html/2508.14925; sha256:e7326aefa910c4da3b99e93cb2f3defdd93d48c927bf7a89ca3d9c54ec1f89c0; archived source capture; exact capture time not independently established
- MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers — AAAI-26 paper · §4.1 Experiment SettingsSelected original resource https://arxiv.org/html/2508.14925; sha256:e7326aefa910c4da3b99e93cb2f3defdd93d48c927bf7a89ca3d9c54ec1f89c0; archived source capture; exact capture time not independently established
- MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers — AAAI-26 paper · §5 DiscussionSelected original resource https://arxiv.org/html/2508.14925; sha256:e7326aefa910c4da3b99e93cb2f3defdd93d48c927bf7a89ca3d9c54ec1f89c0; archived source capture; exact capture time not independently established
Original source charts and materials
Related research questions
References
- [1] Zhiqiang Wang, Yichao Gao, et al.. MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers. 2025-08-19. v2 [arXiv]; Figure 1
- [2] Zhiqiang Wang, Yichao Gao, et al.. MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers — AAAI-26 paper. AAAI-26 proceedings, pp. 35811–35819 ; AAAI-26 original paper, p. 1, author affiliations 1–2