BaLoRA extends LoRA (Low-Rank Adaptation) with a Bayesian parameterization that injects input-adaptive noise into the low-rank updates, providing both uncertainty quantification and improved accuracy. The method adds minimal parameters and compute compared to standard LoRA. Surprisingly, the Bayesian extension not only yields well-calibrated uncertainty estimates but also significantly improves prediction accuracy—narrowing the gap with full fine-tuning—across natural language reasoning, vision tasks, and a scientific prediction task (band gap in metal-organic frameworks). On the science task, BaLoRA's zero-shot uncertainty estimates correlate better with model error than a trained ensemble and improve monotonically with compute.
Main takeaways:
- Standard LoRA uses low-rank point estimates, which limit expressiveness, leave an accuracy gap versus full fine-tuning, and provide no uncertainty.
- BaLoRA adds input-adaptive Bayesian noise to LoRA matrices, providing well-calibrated uncertainty estimates at minimal extra cost.
- The Bayesian noise injection also improves accuracy, narrowing the gap with full fine-tuning across NLP and vision benchmarks.
- On a scientific prediction task (band gap in materials), BaLoRA's uncertainty correlates more strongly with error than an ensemble baseline.
- Uncertainty improves monotonically with compute without sacrificing accuracy, making it suitable for reliability-sensitive applications.