The authors tackle how to monitor deployed healthcare AI models when getting ground-truth labels (like having a clinician manually review charts) is expensive. They extend existing methods that blend a small labeled sample with cheap model predictions by letting you use multiple predictors of different cost and accuracy at once — routing each test case to the right predictor, sampling labels where uncertainty is highest, and reweighting everything to get narrower confidence intervals under a fixed labeling budget. On synthetic and real healthcare tasks they get 10–40% tighter intervals than single-predictor baselines.
Main takeaways:
- Combines adaptive per-instance routing (which predictor to use), smart label sampling (where to spend your annotation budget), and prediction reweighting under one optimization framework.
- Proves the method is globally optimal despite the joint problem being non-convex, and provides valid finite-sample statistical guarantees.
- Particularly useful when you have multiple models of varying cost and accuracy available (e.g., a cheap heuristic and an expensive foundation model) and want to squeeze the most out of a labeling budget.
- Achieves 10–40% narrower confidence intervals than baselines in real healthcare monitoring scenarios.