Skip to content
Sagan

Paper

Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence

Unreadunread

AI summary

arXiv:2606. 12058v1 Announce Type: new Abstract: Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training.