Skip to content
Sagan

Paper

Dynamics of Stochastic Momentum with Sparse Updates in High Dimensions

Unreadunread

AI summary

arXiv:2605. 28961v1 Announce Type: new Abstract: Existing theory of momentum assumes that gradients arrive at every parameter at a roughly constant rate, an assumption violated in practice by heavy-tailed data distributions and modern architectures.