Your AI Adoption Lift Is a Selection Effect

The article explains that measuring the impact of an opt-in AI feature like an AI assistant is complicated by selection bias: accounts that adopt the feature tend to be more engaged and have better retention irrespective of the feature. Naive comparisons overstate the effect due to unobserved engagement confounders. It illustrates three estimation methods using a synthetic dataset with a seat-based eligibility threshold: 1) naive adopter vs. non-adopter retention gap (+15 points, biased), 2) regression adjusting for observed covariates (still biased), and 3) regression discontinuity (RD) at the eligibility cutoff. RD uses the arbitrary seat cutoff as an instrument to isolate causal effects from selection. RD estimates a local average treatment effect of roughly +5 points retention for induced adopters, with wider confidence intervals reflecting true uncertainty. The article highlights diagnostic checks for RD credibility and emphasizes that only variation not chosen by customers (like the cutoff) identifies causal effects. It urges careful interpretation of which estimand fits the business question and warns against overconfident, biased naive estimates.