The author shows that while cosine similarity between pre- and post-SFT activations stays very high (suggesting little change), projecting both through a Sparse Autoencoder reveals that the underlying sparse features diverge significantly. Using SAEs as a diagnostic tool, they identify task-specific and layer-specific distributions of semantic features that are systematically altered during fine-tuning, and discover a layer-wise update profile specific to safety alignment.
Main takeaways:
- High activation cosine similarity after SFT is misleading—the sparse latent structure changes substantially even when dense activations look similar.
- Sparse Autoencoders pretrained on the base model can be used to measure which interpretable features are altered by fine-tuning.
- The changes are task-specific and layer-specific: different fine-tuning objectives modify different semantic features in different layers.
- Safety alignment shows a characteristic layer-wise update profile distinct from other fine-tuning tasks.
- The method provides a high-resolution mechanistic view of what SFT actually changes beneath surface-level geometry.