Skip to content
Sagan

Paper

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably

Unreadunread

AI summary

arXiv:2605. 15514v1 Announce Type: new Abstract: We identify intrinsic limitations of Rotary Positional Embeddings (RoPE) in Transformer-based long-context language models.