The authors introduce C-DSAC, a reinforcement learning algorithm that combines the Soft Actor-Critic approach with distributional RL (learning a full distribution over future rewards instead of just an average). They use the squared Cramér distance to train the model and show it beats standard SAC and other distributional methods on robotics benchmarks, especially in complex environments. The key insight is that C-DSAC does "confidence-driven" updates: when the target distribution has high variance (low confidence), the model makes more conservative updates, reducing the impact of overestimated values.
Main takeaways:
- Represents state-action values as distributions rather than single numbers, capturing uncertainty about future rewards
- Uses the Cramér distance (a way to measure how different two probability distributions are) for learning
- High-variance (uncertain) target distributions automatically lead to smaller, more conservative model updates
- Outperforms baseline SAC and other distributional methods, with the gap widening in high-complexity robotics tasks
- The confidence-driven update mechanism helps prevent overly optimistic value estimates from corrupting learning