The authors built VITA-QinYu, a spoken language model that can do more than just chat — it can role-play different personalities and even sing. The model uses separate tokens for text and audio (with multiple codebooks for richer sound representation) and was trained on nearly 16,000 hours of conversation, role-playing, and singing data. It beats other spoken models both on objective role-playing tasks and on singing quality, while also improving conversational accuracy.
Main takeaways:
- First end-to-end spoken model that handles natural conversation, role-playing (e.g., speaking in a comforting tone), and singing generation in one system
- Uses a hybrid design that keeps text and audio modalities separate to avoid them interfering with each other, while using multi-codebook tokens to capture expressive vocal features
- Outperforms peer models by 7 percentage points on role-playing benchmarks and by 0.13 MOS points (on a 5-point scale) for singing
- Also achieves state-of-the-art on conversational benchmarks, beating prior models by 1.4–5 percentage points
- Code, models, and a streaming full-duplex demo are open-sourced