Skip to content
Sagan

Paper

Why Do Accumulated Transformations Extrapolate?

Unreadunread

AI summary

arXiv:2606. 24975v1 Announce Type: new Abstract: PaTH Attention showed that replacing RoPE's position-indexed rotations with accumulated data-dependent Householder reflections yields strong length extrapolation, though performance degrades at extreme context lengths.