The authors propose Self-Consolidating Language Models (SCoL), a framework that lets an LLM decide which of its own Transformer layers to update when incorporating new context. Instead of simply putting information in the prompt or fine-tuning everything, the model learns to generate "update instructions" specifying sparse layer selections, trained with meta-reinforcement learning so it can adapt as its own weights change. On QA tasks and long-context benchmarks, SCoL beats prompting, summarization, and sequential fine-tuning by learning to route updates toward high-Fisher-information layers (the parts of the model most sensitive to loss) while limiting interference with previously consolidated knowledge.
Main takeaways:
- SCoL lets the model choose which layers to update when incorporating new context, rather than updating all weights or none.
- Trained via meta-RL over an evolving model state—the model that picks future updates is itself being changed by past updates.
- Outperforms prompting, summarization, batch test-time training, and sequential fine-tuning on both SQuAD knowledge incorporation and LongBench v2 long-context tasks.
- Learned update patterns are sparse and align with layers of high Fisher information (regions where small weight changes have big effects on loss).
- Transfers from shorter training streams to longer evaluation streams, suggesting the method scales to streaming contexts.