Skip to content
Sagan

Paper

PowerOPD: Stabilizing On-Policy Distillation with Bounded Power Transformation

Unreadunread

AI summary

arXiv:2606. 17199v1 Announce Type: new Abstract: Standard on-policy distillation (OPD) for large language models estimates the reverse-KL objective using student-sampled tokens, yielding an unbiased single-sample Monte Carlo estimator that avoids vocabulary-wide computation.