The authors built MedSyn, a system where emergency physicians iteratively query an LLM that has the full clinical record while the physician initially sees only the chief complaint. Seven physicians (three seniors, four residents) completed baseline and AI-assisted sessions on 52 MIMIC-IV cases. Blinded evaluation showed residents' correctness on hard cases rose from 58.9% to 73.4%; standardized metrics confirmed medium effect size. Dialogue analysis revealed seniors asked targeted questions while residents asked broader queries, and cross-expertise agreement increased.
Main takeaways:
- Physicians query an LLM with full clinical info while they see only the chief complaint, then iteratively reveal details
- Residents' hard-case correctness improved from 58.9% to 73.4%; difficulty-standardized completely-correct rate showed medium effect (d=0.47)
- Automated metrics: standardized any-match accuracy +15.6 pp, residents' F1 +13.8 pp
- Seniors asked hypothesis-driven questions; residents used broader queries
- Cross-expertise diagnostic agreement increased by 14.5 pp
- Demonstrates measurable benefit in live physician workflows, not just benchmarks