The authors study the information-theoretic limits of learning hierarchical features from a teacher network when the teacher width scales linearly with input dimension (the "extensive-width" regime that captures large-but-finite networks). Using a leave-one-out decoupling argument, they derive equations characterizing the Bayes-optimal generalization error and show that features become learnable through a sequence of sharp phase transitions: as data grows, teacher features are recovered sequentially, each through a discontinuous jump in overlap. This leads to two distinct scaling regimes unified by a single relation involving "effective width" (the number of learnable features at a given data budget).
Main takeaways:
- In the extensive-width regime (teacher width k scales linearly with input dimension d), feature learnability is governed by sharp phase transitions: features become recoverable sequentially through discontinuous jumps in overlap.
- The Bayes-optimal generalization error follows two scaling laws: n^(1/(2β)-1) in the feature-learning regime and n^(-1) in the refinement regime, where β>1/2 is the power-law exponent of the feature hierarchy.
- Both laws collapse to a single relation: ε^BO = Θ(k_c d/n), where k_c is the "effective width" (number of learnable features at data budget n).
- A student trained with Adam near the effective width k_c empirically achieves these optimal scaling laws (up to a small algorithmic gap).
- The framework provides an information-theoretic account of neural scaling laws in model size and data.