The authors built IntentGrasp, a large-scale benchmark for evaluating how well LLMs understand intent in speech, conversation, and writing. Drawn from 49 diverse, open-licensed corpora across 12 domains, the benchmark includes 262k training instances, a 12.9k All Set, and a harder 470-case Gem Set. Testing 20 LLMs (including GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.7) reveals poor performance: below 60% on All Set, below 25% on Gem Set, with 17 of 20 models worse than random guessing (15.2%) on Gem Set, while estimated human performance is ~81%. The authors propose Intentional Fine-Tuning (IFT)—fine-tuning on the training set—which boosts F1 by 30+ points on All Set and 20+ on Gem Set, with strong cross-domain generalization in leave-one-domain-out experiments.
Main takeaways:
- Intent understanding is crucial for helpful assistants but remains unsolved: frontier LLMs score below 60% on All Set and below 25% on the harder Gem Set.
- 17 of 20 tested models perform worse than random guessing on Gem Set; estimated human performance is ~81%, showing huge headroom.
- IntentGrasp unifies 49 corpora across 12 domains into a single benchmark with a large training set (262k) and two test sets (All and Gem).
- Intentional Fine-Tuning (IFT)—simply fine-tuning on the training set—yields 30+ point F1 gains on All Set and 20+ on Gem Set.
- Leave-one-domain-out experiments confirm IFT generalizes well across domains, suggesting it's a promising path to more capable and safer assistants.