Skip to content
Sagan

Paper

A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention

Unreadunread

AI summary

arXiv:2605. 12697v1 Announce Type: new Abstract: Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the context length $n$, ranging from $(\log n)^{1/2}$ to $\log n$ and $(\log n)^2$.