Skip to content
Sagan

Paper

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Unreadunread

AI summary

arXiv:2606. 20945v1 Announce Type: new Abstract: Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length.