The authors argue that group-based RL methods like GRPO suffer from "winner-takes-all" dynamics where rollouts compete for individual advantage, causing models to converge on narrow high-scoring patterns. They propose GCPO, which replaces individual scoring with team-level credit assignment: a rollout is rewarded based on how much it contributes to the team's coverage of valid, non-redundant solutions (measured as a determinant volume over reward-weighted semantic embeddings). This cooperative paradigm routes optimization toward diverse correct reasoning paths, improving both accuracy and solution diversity on reasoning benchmarks.
Main takeaways:
- Standard group RL (like GRPO) creates competition among rollouts, leading to premature convergence on a narrow set of solutions.
- Adding entropy bonuses or diversity rewards doesn't fix the core problem because rollouts still compete rather than cooperate.
- GCPO rewards rollouts based on their marginal contribution to the team's collective coverage of correct, distinct solutions.
- Coverage is computed as a volume in semantic embedding space, weighted by correctness—only non-redundant correct answers contribute.
- Experiments show GCPO improves both reasoning accuracy and solution diversity compared to existing methods.