The authors develop a framework for valid statistical inference when data are collected via active data collection (ADC)—e.g., Bayesian optimization or sequential model-based optimization—and then reused for a post-hoc inferential task (like testing whether a discovered setting is truly optimal). Standard inference fails because ADC preferentially samples regions the algorithm thinks are good, creating adaptive bias. The "post-ADC inference" framework corrects for both the bias from adaptive data collection and the bias from constructing the inferential target in a data-dependent way, providing valid p-values and confidence intervals.
Main takeaways:
- Active data collection (e.g., Bayesian optimization) adaptively biases sampling toward promising regions, breaking standard statistical inference.
- Standard p-values and confidence intervals are invalid when you reuse ADC data for post-hoc inference (e.g., "is this the true optimum?").
- Post-ADC inference corrects for both the adaptive sampling bias and the data-dependent construction of the inferential target.
- The method builds on selective inference and applies to a broad class of ADC processes (only assumes observation noise, not the black-box function or surrogate model).
- Empirical results show valid inference for data collected by GP-UCB and tree-structured Parzen estimator (TPE) with correct coverage.