The authors introduce AcuityBench, a benchmark that tests whether language models correctly judge how urgently a patient needs care (home monitoring, scheduled visit, urgent care, or ER) from medical descriptions. It pools 914 cases from five datasets (conversations, forum posts, clinical vignettes, patient messages), including 217 ambiguous cases where physicians themselves disagreed. They test 12 frontier models in both QA format (pick one of four urgency levels) and free-form conversational format, finding substantial variation in accuracy and a systematic tradeoff: conversational responses reduce over-triage but increase under-triage, especially for high-acuity cases. No model closely matches the distribution of physician uncertainty on ambiguous cases.
Main takeaways:
- AcuityBench unifies five public medical datasets under a shared four-level urgency framework (home monitoring → scheduled → urgent → ER) with 697 clear-consensus cases and 217 physician-confirmed ambiguous cases.
- Frontier models vary widely in accuracy and error direction; some over-triage, others under-triage.
- Conversational responses systematically reduce over-triage but increase under-triage compared to QA format, especially for high-acuity cases.
- No model's uncertainty distribution on ambiguous cases matches physician judgment distributions — models are more confident and concentrated.
- Acuity identification is a distinct safety-critical capability separate from general medical QA.