The paper extends multicalibration (requiring predicted scores to match true label probabilities across many subgroups and score-dependent tests) to weakly supervised learning settings where clean labels are unavailable—like positive-unlabeled learning or noisy-label scenarios. Existing multicalibration methods assume clean input-label pairs for evaluation and correction, which doesn't hold in these regimes. The authors develop estimators of multicalibration error and post-hoc correction methods for weak supervision by combining contamination-matrix risk corrections with witness-based calibration constraints, yielding corrected multicalibration moments with finite-sample guarantees. They propose WLMC, a generic recalibration algorithm for weak supervision.
Main takeaways:
- Extends multicalibration (calibration across rich subgroup families) to weakly supervised settings without clean labels
- Combines contamination-matrix risk rewrites with witness-based calibration to estimate and correct multicalibration error under weak supervision
- Provides finite-sample guarantees for the corrected multicalibration estimators
- Proposes WLMC, a generic post-hoc recalibration algorithm, with experiments across multiple weak-supervision scenarios