The authors introduce MIST, a synthetic benchmark for voice-driven smart-home assistants that combines speech input, tool-calling over IoT devices, and mixed-initiative multi-turn dialogue. The task requires models to generate code that respects spatiotemporal constraints (e.g., "turn off the lights in the kitchen but only if no one is there"), track dynamic device state across turns, and handle interruptions or clarifications from the user. Benchmarking open- and closed-weight multimodal LLMs reveals a large gap: even frontier closed models have substantial room for improvement. The dataset and generation framework are released to support research on voice assistants that reason about the physical world.
Main takeaways:
- MIST combines speech inputs, multi-turn dialogue, tool-calling (code generation for IoT devices), and spatiotemporal reasoning in one benchmark.
- The task is synthetic but designed to reflect real smart-home complexity: dynamic state, mixed initiative (user can interrupt or clarify), and physical-world constraints.
- Open-weight multimodal LLMs lag far behind closed-weight models; even the best closed models have significant headroom.
- The authors release both the dataset and an extensible data-generation framework so others can create similar benchmarks for related domains.
- Key challenge: modeling physical-world constraints ("Is anyone in the room?") alongside traditional NLP reasoning.