The authors develop theory for learning multiple independent components simultaneously in high-dimensional ICA, going beyond prior work that only analyzed single-component recovery. They show that in the high-dimensional limit, the learning dynamics follow a deterministic ODE for the overlap between learned and true components. This reveals two regimes: a "decoupled" regime where estimates align cleanly with distinct components, and a "competition" regime where overlapping initializations cause conflicts, slow reorientation, and delayed convergence.
Main takeaways:
- Multi-component ICA has richer dynamics than single-component recovery because learning multiple directions simultaneously creates coupling through orthogonalization
- In high dimensions, the joint distribution of learned and true components converges to a deterministic process described by an ODE for the overlap matrix
- Two regimes emerge: decoupled (estimates align with distinct components and evolve independently) and competition (overlapping initializations induce conflicts and slow convergence)
- Larger higher-order moments and initialization overlap shrink the stable learning-rate window and increase convergence time
- Predicts a "staircase" phenomenon where the number of recoverable components changes discretely with learning rate