Skip to content
Sagan

Paper

Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Models

Unreadunread

AI summary

arXiv:2606. 25086v1 Announce Type: cross Abstract: Many modern Language Model (LM) pipelines return an averaged model, such as an exponential moving average of the training iterates, rather than the final iterate itself.