The authors investigate when on-policy distillation (OPD, where a student learns from a teacher on the student's own outputs) and on-policy self-distillation (OPSD, where a model learns from itself conditioned on extra information) succeed or fail. They find OPD on math reasoning is fragile and sensitive to teacher choice and loss function, while OPSD fails when the extra "privileged information" (like a hint) is unique to each problem but works when it's a shared rule (like a system prompt or alignment preference). Three failure mechanisms: (1) teacher-student distribution mismatch from conditioning on student prefixes, (2) optimization instability from biased gradients in TopK reverse-KL, and (3) OPSD can't learn instance-specific hints because it averages over hint-conditioned teachers. Fixes include stop-gradient objectives, adapted teachers, and SFT stabilization.
Main takeaways:
- OPD (student learns from teacher on student's own rollouts) is unstable for math reasoning—highly sensitive to which teacher you pick and which loss you use.
- OPSD (model learns from itself + extra info) works when the info is a general rule (system prompt, alignment preferences) but fails when the info is problem-specific (instance hints).
- The OPSD failure on instance-specific info happens because the student learns a single policy that averages over all hint-conditioned behaviors, which doesn't help at test time when you don't have the hint.
- Distribution mismatch (teacher sees student prefixes it wouldn't generate itself) and biased gradients (TopK reverse-KL) cause training instability.
- Mitigation strategies: stop-gradient TopK objectives, adapting the teacher with RLVR, and stabilizing the student with supervised fine-tuning first.