The authors propose Toeplitz MLP Mixers (TMMs), a Transformer alternative that replaces attention with multiplication by Toeplitz matrices (structured matrices where diagonals are constant). This drops computational complexity from quadratic to O(n log n) during training and linear at inference, while using less memory. Despite lacking the sophisticated gating and state tracking of architectures like Mamba, TMMs retain more information from inputs, showing better copying ability and in-context learning. The authors argue this comes from fewer architectural biases, and they find that trained Toeplitz layers tend to be nearly invertible even though the overall model isn't required to be.
Main takeaways:
- Toeplitz matrix multiplication (where each diagonal has the same value) replaces quadratic-cost attention with O(n log n) training and O(n) inference complexity
- TMMs lack explicit gating or state maintenance but achieve better training efficiency (lower loss per compute and memory usage) than comparable architectures
- Higher information retention from inputs leads to improved copying, retrieval, and in-context learning compared to other sub-quadratic alternatives
- Counterintuitively, trained Toeplitz layers are more likely to be invertible (or nearly so) than in architectures designed to be invertible
- The authors argue fewer architectural biases allow the model to retain more input information, benefiting tasks requiring faithful recall