Skip to content
Sagan

Paper

Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Test

Unreadunread

AI summary

arXiv:2606. 20743v1 Announce Type: new Abstract: Trained transformers reliably develop massive activations, a small number of hidden dimensions whose magnitude is far above the median and which concentrate on the sequence-start token.