Laplace approximation provides a way to quantify uncertainty in neural networks by approximating the posterior with a Gaussian centered at the MAP estimate, but inverting the full Hessian is computationally prohibitive. Sub-network Laplace methods restrict attention to a subset of parameters to make it tractable. This paper proves that all such methods systematically underestimate predictive variance and that the bias decreases monotonically as you include more parameters. They propose two principled subset selection methods: Gradient-Laplace (selects parameters with largest average squared output gradients) and Greedy-Laplace (iteratively adds parameters accounting for correlations). They prove Gradient-Laplace provably beats existing heuristics and demonstrate strong empirical performance.
Main takeaways:
- Sub-network Laplace restricts the Hessian to a parameter subset to make uncertainty quantification tractable
- All such methods underestimate predictive variance; bias decreases monotonically as the subset grows
- Gradient-Laplace selects parameters with largest average squared gradients of model output (provably optimal in a certain sense)
- Greedy-Laplace iteratively refines selection by accounting for off-diagonal Hessian terms
- Extensive experiments show these methods outperform existing heuristics (diagonal, layer-wise, etc.)