The authors identify that diffusion language models (an alternative to autoregressive LMs) have redundant attention activations across tokens during inference. They built DARE, which reuses cached key-value pairs (DARE-KV) and output activations (DARE-O) when the model's internal state predicts they won't change much. This achieves up to 1.20x per-layer speedup and reuses up to 87% of attention activations with minimal quality loss (around 2% average performance drop). The technique is compatible with other speedup methods like prefix caching.
Main takeaways:
- Diffusion LLMs show high token-wise redundancy in bi-directional self-attention activations
- DARE reuses cached key-value and output activations when temporal changes in queries predict redundancy
- Achieves up to 1.20x per-layer latency reduction, reusing up to 87% of attention activations
- Average performance drops are only 2.0% (DARE-KV) and 1.2% (DARE-O) on reasoning and code benchmarks
- Compatible with other inference optimizations like prefix caching and Fast-dLLM