Skip to content
Sagan

Paper

Blurry Window Attention

Unreadunread

AI summary

arXiv:2606. 09862v1 Announce Type: new Abstract: The Softmax Attention operation in Transformer language models has a quadratic complexity in the sequence length and a growing state size in the form of KV cache, which becomes a bottleneck in long context scenarios.