The authors present LatentRouter, a router that picks which multimodal LLM to use for a given image-question query before you see any model's answer. Instead of just estimating query difficulty, it predicts how well each candidate model would perform by extracting "multimodal routing capsules" from the input, representing each model with a learned capability token, and simulating latent communication between them to estimate counterfactual utility. On MMR-Bench and VL-RouterBench, LatentRouter outperforms fixed-model and other learned-router baselines, especially on tasks where visual, layout, or reasoning requirements differ across models.
Main takeaways:
- Frames multimodal model routing as counterfactual utility prediction: estimate how each model would perform if you picked it, without actually running it.
- Extracts learned multimodal capsules from the input and uses latent communication with model capability tokens to predict performance.
- Supports both performance-only and performance-cost routing, and can handle changing candidate pools via shared per-model scoring.
- Strongest gains come on tasks where model choice depends on visual layout or reasoning mode, not just query difficulty.
- Latent communication (the interaction between input capsules and model tokens) is the main source of improvement over simpler feature-based routing.