The authors built a pipeline to distill reasoning ability from large vision-language models into smaller ones that are fast enough to deploy in production. They curated 1.8M high-quality chain-of-thought examples from teacher models, filtered them by difficulty and diversity, then used them to train smaller Qwen3-VL models (2B-8B parameters). The distilled 4B model matched or beat the undistilled 8B baseline on several vision-reasoning benchmarks, showing you can compress reasoning capability without losing much performance.
Main takeaways:
- Distilling chain-of-thought reasoning from big multimodal models to small ones makes deployment practical without sacrificing much accuracy
- Their 1.8M-sample dataset includes difficulty ratings and task tags, letting you control what kind of reasoning examples you train on
- A 4B distilled model matched an 8B undistilled model on several tasks, and improved +16.8 points on MathVerse
- The pipeline combines rule-based filtering, difficulty-aware sampling, and semantic tags to maintain quality at scale
- Shows that lack of high-quality CoT supervision is the main bottleneck for smaller models, not inherent capability limits