The authors build a Transformer variant that keeps intermediate representations on a curved geometry (a manifold) using rotation and reflection operations that preserve distances and angles. The key innovation is a way to combine two different mathematical tricks (Cayley transforms for rotation, Householder reflections for flipping) with a learned gate that decides which operation to use, ensuring the network never accidentally distorts its representations. They show this hybrid approach gives better stability during training and handles both types of orthogonal transformations (rotations and reflections) where previous methods could only do one or the other.
Main takeaways:
- Orthogonal operations (transformations that preserve vector lengths and angles) help neural networks stay stable during training, especially over long sequences
- Standard methods like Deep Delta Learning only guarantee orthogonality at specific settings; this work maintains it for all parameter values by using Cayley transforms
- A learned gate decides whether to rotate or reflect representations at each layer, with a penalty encouraging clean either/or choices rather than blending
- In experiments with ~1.8M parameters, the architecture showed 1.9× better long-term stability than a competing method and 3.8× better than GPT-style baselines
- The method needs 33% fewer layers to achieve similar performance to baselines