MCPShield is an attack-detection system for LLM agent tool-call traffic using the Model Context Protocol. It represents each agent session as a graph (tool calls as nodes, sequential and data-flow links as edges), enriches nodes with sentence embeddings of arguments and responses, and classifies sessions as benign or attacked. The key finding is that content-level features (embeddings of what the agent actually said and received) are essential—metadata alone plateaus around 64% AUROC while content embeddings push it above 89%.
Main takeaways:
- Metadata-only detection (just tracking which tools were called, in what order) achieves only ~64% AUROC regardless of architecture; content embeddings are necessary for good performance
- Naive random train-test splits inflate AUROC by up to 26 percentage points compared to task-disjoint splits—a memorization problem that prior agent-detection work hasn't addressed
- Tree ensembles on pooled SBERT embeddings reached 97.5% AUROC, outperforming GNNs (91.7%) and MLPs (89.6%) in the primary evaluation
- The detection signal lives primarily in the semantic content embeddings, not in the graph structure or tool-call metadata