Most knowledge distillation for LLMs uses only the teacher's output logits, ignoring intermediate layer representations. The authors test Hidden Layer Distillation (HLD)—matching student hidden states to teacher hidden states—during decoder-only pretraining at scale (up to 168B tokens, Gemma3 3.4B teacher, 123M and 735M students). HLD consistently lowers perplexity compared to standard logit-based distillation, but doesn't reliably improve downstream task performance, suggesting the signal is there but not yet actionable for real-world use.
Main takeaways:
- Hidden Layer Distillation matches student intermediate representations to teacher representations, not just output logits
- Tested at scale: up to 168B tokens from C4, teacher is Gemma3 3.4B, students are 123M and 735M
- HLD systematically reduces perplexity versus logit-based distillation across all configurations
- But HLD doesn't consistently beat logit-based KD on downstream task benchmarks—perplexity gains don't transfer to performance