The authors develop an active inference framework for U-statistics (a class of estimators central to many statistical tests) when labels are expensive. Instead of collecting all labels, they selectively query the most informative ones to improve estimation efficiency under a fixed budget while preserving valid statistical inference. The method uses augmented inverse probability weighting to account for the adaptive sampling rule and machine learning predictions, characterizes the optimal sampling rule that minimizes variance, and extends to empirical risk minimization based on U-statistics.
Main takeaways:
- U-statistics (estimators based on averaging over all subsets of data) are fundamental in statistics but often require expensive labels in modern applications.
- Active inference selectively queries labels to maximize information gain under a fixed labeling budget.
- The method uses augmented inverse probability weighting to incorporate the adaptive sampling rule and ML predictions, ensuring valid inference.
- The authors derive the variance-minimizing optimal sampling rule and provide practical sampling strategies.
- The framework extends to U-statistic-based empirical risk minimization (e.g., AUC optimization).
- Experiments show substantial efficiency gains over baseline methods while maintaining correct coverage.