The authors tested whether generating multiple solution approaches for math problems during an intermediate training stage ("mid-training") improves reinforcement learning afterward. They used a bootstrapped self-generation method inspired by Polya's problem-solving framework to create diverse correct answers for each training question, fine-tuned on that data, then ran standard RL. Models initialized with this mid-training consistently outperformed baselines on math reasoning benchmarks and generalized better to out-of-distribution tasks like code generation. The paper argues theoretically that policy-gradient RL can learn to combine multiple solution strategies when the model has been exposed to them during mid-training.
Main takeaways:
- Mid-training on self-generated diverse solutions (multiple valid approaches per problem) before RL improves downstream RL performance on math reasoning
- The diversity comes from generating solution variants guided by different problem-solving strategies (Polya's framework)
- RL-trained models initialized with this mid-training data outperform baselines on multiple math benchmarks and generalize to code generation and narrative reasoning
- The authors provide a theoretical argument that policy-gradient updates can learn to combine approaches when the model has seen multiple valid paths during mid-training
- This is distinct from standard supervised fine-tuning or RL from scratch: the intermediate self-generated diversity stage is the key ingredient