The paper studies three ways to make neural networks respect symmetries in data (e.g., rotating a point cloud shouldn't change the prediction): building invariance into the architecture, averaging over transformations, or "canonizing" (sorting or otherwise standardizing) inputs before feeding them to a non-invariant backbone. The authors prove that canonization's generalization depends on how smooth the canonization function is — a bad canonization can be as poor as no symmetry handling at all, while a good one matches fully invariant architectures. They show that lexicographic sorting of point clouds has exponentially large covering numbers (bad), whereas Hilbert-curve sorting has polynomial covering numbers (good), explaining why Hilbert serialization works well in practice.
Main takeaways:
- Canonization (preprocessing to fix symmetry before a standard model) isn't automatically as good as building invariance into the network.
- The authors bound generalization error via covering numbers and show canonization sits between fully invariant models (best) and non-invariant baselines (worst).
- Whether canonization reaches the good end of that spectrum depends on the regularity (smoothness) of the canonization map.
- For point clouds, lexicographic sorting has exponential covering-number growth, but Hilbert curve canonization has polynomial growth — the first rigorous theory for why Hilbert serialization helps.
- Experiments on point-cloud tasks confirm the theoretical hierarchy.