The authors propose RankQ, an offline-to-online reinforcement learning method that improves Q-learning by adding a self-supervised ranking loss. Instead of uniformly penalizing out-of-distribution actions (the standard pessimism approach), RankQ learns relative preferences between actions, shaping the Q-function so gradients point toward better behaviors. This helps when the offline dataset contains suboptimal actions—prior methods anchor too hard to dataset actions and limit downstream improvement. On D4RL sparse-reward benchmarks and vision-based robot learning, RankQ matches or beats prior methods, achieving 42.7% higher success in low-data VLA fine-tuning and 13.7% improvement in high-data settings, with strong sim-to-real transfer (cube stacking success from 43% to 85%).
Main takeaways:
- Standard offline RL methods impose pessimism by down-weighting unseen actions, which acts like behavior cloning and limits improvement when the dataset is suboptimal.
- RankQ adds a multi-term ranking loss to TD learning, teaching the Q-function to order actions by quality rather than just penalizing anything not in the dataset.
- This directs action gradients toward higher-quality behaviors, enabling better online fine-tuning from limited data.
- Competitive or superior to seven prior methods on D4RL benchmarks; particularly strong in vision-based robot learning with pretrained VLA models.
- Achieves strong real-world transfer, nearly doubling cube-stacking success rate over the initial VLA policy.