SOMA tackles the cost of multi-turn LLM conversations by switching from a large model to a smaller surrogate after a few turns. The idea: use early turns to map out the "response manifold" (the space of likely responses for this conversation), then fine-tune a small model with LoRA specifically for that local region. They learn soft prompts that maximize divergence between large and small model responses to find where alignment is weakest, then distill those cases into a LoRA adapter so the small model runs without extra prompt overhead at inference.
Main takeaways:
- Standard multi-turn serving concatenates full history every turn, which is expensive in latency, memory, and API cost.
- SOMA uses early turns to identify a "local response manifold" and adapts a small model to handle the rest of the session.
- They mine hard cases by maximizing semantic divergence between large and small model outputs, then distill into LoRA fine-tuning.
- A gating mechanism enables one-time switching with rollback if the conversation drifts outside the learned region.
- Experiments show efficiency gains while maintaining response quality compared to always using the large model.