The authors propose GLoRA, a federated-learning approach for LoRA fine-tuning that fixes a conceptual problem: directly averaging LoRA factors (the A and B matrices) is "gauge-dependent"—you can rotate or rescale them arbitrarily without changing the actual update, so factor averaging can produce different results depending on the coordinate system. GLoRA instead estimates a consensus update subspace from clients' projectors and aggregates updates in shared coordinates, making aggregation semantically meaningful. It also supports heterogeneous client ranks (different clients can use different LoRA ranks) by projecting onto a shared server representation.
Main takeaways:
- Standard federated LoRA averages factors directly, but this is coordinate-dependent—the same update can be written infinitely many ways
- GLoRA estimates a consensus subspace and aggregates updates in shared reference coordinates, making aggregation gauge-invariant
- Supports heterogeneous client ranks: clients can use different LoRA ranks and GLoRA projects them onto a shared server state
- Outperforms federated LoRA baselines on GLUE and SuperNI under data, resource, and task heterogeneity