Full fine-tuning (FFT) gives models the plasticity to learn high-entropy new knowledge, while LoRA often matches or beats FFT because many tasks only need low-rank updates and benefit from LoRA's regularization. The authors propose MoLF, which dynamically routes gradient updates between FFT and LoRA experts at the optimizer level during training, letting the model use whichever is better for each update. Across SQL, medical QA, and counterfactual knowledge tasks, MoLF matches or beats the better of FFT or LoRA by up to 1.5%. A memory-efficient variant (MoLF-Efficient) freezes base weights and routes between two LoRA experts of different rank.
Main takeaways:
- Full fine-tuning provides plasticity for complex updates; LoRA provides regularization and works when updates are low-rank
- Statically choosing one method is suboptimal—different tasks and training steps benefit from different regimes
- MoLF routes updates between FFT and LoRA experts dynamically at the optimizer level, getting the best of both
- MoLF-Efficient (LoRA-only routing) outperforms prior adaptive LoRA methods by up to 20% on factual tasks