Skip to content
Sagan

Paper

More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Unreadunread

AI summary

arXiv:2605. 26647v1 Announce Type: cross Abstract: Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs).