The authors tackle a common problem in fine-tuning: when you train a language model on one task, it gets better at that task but worse at unrelated things. They propose RPSFT, a method that penalizes changes to the dominant directions in the pretrained weight matrices (the "top-k singular vectors") during fine-tuning, treating this as a cheaper proxy for Fisher-sensitive directions. The idea is to let the model adapt without breaking the pretrained representation structure that supports general capabilities.
Main takeaways:
- Standard supervised fine-tuning (SFT) damages out-of-domain performance, and prior work links this damage to rotation of the dominant singular subspaces in weight matrices.
- RPSFT penalizes changes in the projected top-k singular-vector block of each pretrained weight, limiting rotation while still allowing task adaptation.
- On math reasoning datasets across multiple model sizes, RPSFT improves the in-domain vs. out-of-domain trade-off compared to standard SFT and strong baselines.
- The method better preserves pretrained representations and provides stronger initializations for downstream reinforcement learning fine-tuning.
- The approach is computationally cheaper than directly computing Hessian or Fisher information at LLM scale.