Skip to content
Sagan

Paper

Repetition Mismatch: Why Data Mixture Experiments Don't Scale and How to Fix Them

Unreadunread

AI summary

arXiv:2606. 07597v1 Announce Type: new Abstract: Pre-training data mixtures are commonly tuned by running small-scale experiments and extrapolating to the target training budget.