Skip to content
Sagan

Paper

Defense effectiveness across architectural layers: a mechanistic evaluation of persistent memory attacks on stateful LLM agents

Unreadunread

AI summary

This paper systematically tests six defenses against "persistent memory attacks" on LLM agents — attacks where malicious instructions injected via RAG-retrieved documents get stored in the agent's memory and execute in later sessions. The author ran 5,040 experiments across nine open-source models and found that four defenses completely fail (input filtering and retrieval filtering all achieve ~88% attack success, same as no defense). Prompt hardening partly works but mostly due to model-level refusal. Only one defense, Memory Sandbox (blocking the agent's ability to recall from memory), reduces attacks to 0% for eight of nine models — though it backfires on one reasoning model that inverts to 100% attack success.

Main takeaways:

  • Four common defenses (input sanitizers, retrieval filters, RAG judges) fail because they can't see or block semantically-masked malicious content
  • The architectural explanation: input-level defenses can't observe RAG content; retrieval classifiers are defeated by compliance-framed instructions
  • Memory Sandbox works by removing the recall capability the attack requires, achieving 0% attack success on most models
  • One reasoning model inverts under Memory Sandbox: it refuses attacks naturally (0% ASR) but hits 100% ASR when forced onto the RAG pathway
  • Memory Sandbox has zero utility cost when there's no attack (100% benign task completion)