The authors diagnose why Proximal Policy Optimization (PPO) underperforms in multi-task reinforcement learning: the critic (value function) side has gradient ill-conditioning, meaning easy tasks dominate updates and hard "tail" tasks stall. They propose TOPPO, which adds "Critic Balancing" modules to fix gradient conditioning and balance learning dynamics across tasks, achieving better mean and tail-task performance than SAC-based methods on Meta-World+ while using fewer parameters and environment steps.
Main takeaways:
- PPO struggles in multi-task RL because critic-side gradients are ill-conditioned: easy tasks dominate value updates, starving hard tasks.
- TOPPO reformulates PPO with modules that improve gradient conditioning and balance learning across tasks.
- It matches or beats strong SAC baselines early in training and maintains superior performance at full training budget.
- Uses substantially fewer parameters and environment steps than SAC-family and ARS-family baselines.
- Demonstrates that on-policy methods can rival off-policy approaches in multi-task RL with proper optimization fixes.