The authors develop a theoretical framework for understanding how Transformers approximate functions. Their key insight is that Transformers can build local approximations of a target function and blend them together using softmax as a "partition of unity"—the attention mechanism creates spatial localization and softmax stitches the pieces into a coherent global output. They prove that shallow-but-wide Transformers with just two encoder blocks can approximate smooth functions efficiently and achieve near-optimal generalization.
Main takeaways:
- Transformers work by learning many local approximations and using softmax attention to weight-average them based on position
- Two encoder blocks plus simple feed-forward layers are enough to uniformly approximate smooth (Hölder continuous) functions with O(ε^(-d/α)) parameters
- Generalization error is near minimax-optimal at O(n^(-2α/(2α+d)) log n) for n training samples
- The architecture studied is shallow and wide (not deep), uses softmax and sinusoidal positional encodings like real Transformers
- Softmax plays a dual role: it's both the aggregation mechanism and the key to proving uniform approximation bounds