The authors created VLATIM, a benchmark using the classic puzzle game The Incredible Machine 2 to test whether vision-language models can do human-like logical problem-solving that requires both planning and precise mouse control. The benchmark has five difficulty levels, from basic visual recognition to full puzzle solving. Results show a big gap: large proprietary models can reason about what to do but struggle with precise visual grounding (e.g., clicking the right spot), so they don't yet match human-like problem-solving.
Main takeaways:
- Existing VLM benchmarks skip the hard part: translating high-level reasoning into continuous, precise actions (like point-and-click).
- The benchmark tests five capabilities: visual grounding, domain understanding, object manipulation, multi-step tasks, and full puzzle solving.
- Big models plan well but fail at execution—they can't reliably ground their plans in precise visual coordinates.
- The reasoning-execution gap is the main bottleneck preventing human-like performance on interactive tasks.
- Physics puzzle games expose failure modes that simpler benchmarks miss.