The authors propose PRISM, a method to diagnose how post-training modifications (quantization, LoRA, distillation) change an LLM, not just whether they degrade it. They derive a mathematical upper bound on the quality gap between original and modified models by exploiting the linear output head and near-isometric backbone structure, then decompose the drift into three independent dimensions: scale mismatch, shape distortion, and head divergence. Each dimension corresponds to a different failure mode and suggests specific fixes, and the shape term can be used as a regularizer during training to prevent forgetting.
Main takeaways:
- Existing similarity metrics (CKA, SVCCA) flag degradation but don't explain what went wrong or suggest remedies
- PRISM decomposes model drift into three axes: scale (magnitude mismatch), shape (geometric distortion), and head (output-layer divergence)
- Each axis maps to specific problems: shape breaks under low-bit quantization, scale separates under LoRA forgetting, head diverges under k-quantization
- Ranks model variants with ~0.82 Spearman correlation to actual performance, helping choose which variant to deploy
- The shape term is differentiable and works as a training regularizer, outperforming experience replay at preventing catastrophic forgetting