The authors tackle a problem in federated learning (training models across distributed data): when you combine parameter-efficient fine-tuning like LoRA (which trains only small adapter modules) with differential privacy (adding noise to protect data), the added noise often overwhelms the small weight updates and kills accuracy. Their solution, FedPower, avoids perturbing the mismatched low-rank pieces directly. Instead, the server reconstructs full-rank updates, clips them, aggregates them exactly, then uses a new method called PowerDP to project back into low-rank space while injecting privacy noise in a way that preserves matrix structure.
Main takeaways:
- Standard LoRA with differential privacy suffers because noise drowns out the small adapter weight signals
- FedPower reconstructs full-rank updates before adding noise, avoiding early aggregation errors
- PowerDP injects differential-privacy noise during subspace iteration before orthonormalization, which preserves useful matrix structure
- Achieves both sample-level and client-level differential privacy with rigorous bounds
- Experiments on language tasks show FedPower is robust even with tight privacy budgets and minimal computational overhead