This paper introduces CDLinear, a circulant-structured neural network layer inspired by a physics framework, that diagonalizes the training Hessian (the matrix of second derivatives of the loss) via discrete Fourier transform. The result is near-perfect conditioning (eigenvalue ratios close to 1) and 3.8× fewer parameters than a dense MLP for comparable accuracy on MNIST. The construction gives provable bounds on the Hessian condition number and specifies a principled dropout rate from a physics-calibrated "noise rate."
Main takeaways:
- CDLinear layers are block-circulant, so their Hessian can be FFT-diagonalized, yielding eigenvalues you can read off from input statistics.
- Under whitened inputs, the population Hessian condition number is exactly 1; empirically it stays very low even on finite samples.
- A CDLinear MLP with 2,380 parameters achieves 97.5% on MNIST vs. 98.15% for a dense MLP with 8,970 parameters, and the CD-MLP's Hessian is 310× better conditioned.
- The method transfers a physics-derived "Shannon noise rate" to set dropout, offering a non-arbitrary hyperparameter choice.