The authors introduce CERSA, a memory-efficient fine-tuning method that compresses pretrained models by keeping only the most important 90–95% of their "spectral energy" (think of it as the strongest patterns in the model's weights, identified via SVD—singular value decomposition). Instead of storing all frozen weights like LoRA does, CERSA throws away the weak directions and fine-tunes low-rank updates on the remaining principal subspace, cutting memory use while beating standard parameter-efficient methods across vision, text-to-image, and language tasks.
Main takeaways:
- CERSA uses SVD to discard the low-energy components of pretrained weights, keeping only the top principal components that capture 90–95% of the signal.
- This lets you fine-tune with much lower memory than LoRA or other PEFT methods, because you don't store the full frozen model.
- Empirical tests on image recognition, text-to-image generation, and NLU show CERSA matches or beats state-of-the-art PEFT while using substantially less memory.
- The method addresses a core LoRA limitation: low-rank updates miss the actual rank structure of full fine-tuning, creating a performance gap.