Skip to content
Sagan

Paper

Large-Step Training Dynamics of a Two-Factor Linear Transformer Model

Unreadunread

AI summary

arXiv:2605. 21292v1 Announce Type: new Abstract: Gradient-flow analyses show that simplified linear transformers can learn the in-context linear-regression algorithm, but they do not explain the finite-step behavior of gradient descent at large learning rates.