Machine learning models trained on complex health surveys (like NHANES) typically ignore survey design features—primary sampling units, stratification, and sampling weights—which violates independence assumptions and leads to biased estimates, underestimated uncertainty, and misleading fairness assessments. The authors propose Survey-aware Machine Learning (SaML), a nine-step guideline that incorporates survey metadata throughout the ML pipeline. They conduct a scoping review of 16 methodological papers covering weighted training, design-based cross-validation, and survey-adjusted evaluation, and identify gaps in hyperparameter tuning and deployment. They provide a task-specific checklist clarifying which steps are needed for different analytical goals to ensure valid population-level inference.
Main takeaways:
- Standard ML on survey data ignores sampling design (weights, strata, clusters), causing bias and invalid uncertainty estimates
- SaML provides a nine-step guideline integrating survey metadata across the ML lifecycle
- Scoping review of 16 papers summarizes methods for weighted training, design-based CV, and survey-adjusted metrics
- Identifies gaps in hyperparameter tuning and deployment under complex survey designs
- Provides task-specific checklists for valid population inference from survey data