This paper proposes a modification to Group Relative Policy Optimization (GRPO) that addresses training instability by automatically down-weighting extreme token-level updates using a Gaussian kernel. The method is motivated by the theoretical relationship between entropy changes and the covariance between token probabilities and advantages, and requires no additional hyperparameters. Experiments show improved reasoning performance and more stable entropy compared to standard GRPO.
Main takeaways:
- GRPO struggles with exploration-exploitation tradeoffs, leading to suboptimal performance and training instability
- The proposed covariance-weighted method uses a Gaussian kernel to automatically reduce extreme token updates without manual hyperparameter tuning
- Improves downstream reasoning benchmark performance while stabilizing entropy throughout training
- Based on theoretical insight that entropy changes are governed by covariance between token probabilities and advantages