The author demonstrates that fitting behavioral curves (like engagement vs. exposure count) to aggregated data produces systematically misleading results due to Simpson's paradox. On Goodreads data from 3.3M users across 9 genres, individual users peak at around 11 exposures while the aggregate curve peaks at 34—a 3x distortion caused by survival bias (users who stay longer accumulate more exposures and look more engaged). Amazon Electronics shows a 5.3x distortion, while MovieLens (low attrition) serves as a negative control confirming survival bias is the culprit. The author introduces Synthetic Null Calibration to reduce false positives in per-user classification by 32%.
Main takeaways:
- Aggregated behavioral curves can be wildly misleading: Goodreads shows a 3x gap (11 vs 34 exposures at peak), Amazon Electronics 5x, driven by survival bias
- MovieLens (low attrition) shows minimal distortion, confirming survival bias—not aggregation itself—is the mechanism
- The distortion is robust across different ways of slicing categories, measuring engagement, and calibrating classifiers
- Synthetic Null Calibration reduces a 32% false positive rate in per-user behavioral classification