Deciding which layers to freeze or train during continued pretraining is usually a black-box empirical decision. The authors introduce LayerTracer, a diagnostic framework that tracks where in the network task execution happens and how sensitive each layer is to updates. Analysis shows deep layers are where task execution occurs and are highly stable, while shallow layers are more sensitive. Guided by this, they run controlled experiments showing that training shallow layers while freezing deep layers consistently beats full-parameter fine-tuning and the opposite allocation on Chinese benchmarks (C-Eval, CMMLU). A hybrid model case study confirms that placing high-quality pretrained modules in deep layers preserves inherent knowledge.
Main takeaways:
- LayerTracer reveals task execution positions and layer sensitivity to updates in an interpretable way
- Deep layers handle task execution and are stable; shallow layers are more sensitive to updates
- Training shallow + freezing deep layers outperforms full fine-tuning and the reverse strategy on Chinese evals
- Hybrid models benefit from placing high-quality modules in deep layers to preserve knowledge
- Provides actionable, low-cost guidance for resource-constrained continued pretraining