You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?

Question

RED SAG · Accepted Answer

Propensity methods only adjust for confounders we measure, so the biggest risk is unmeasured ones like motivation, which drives both adoption and retention. I would list likely confounders with the product team and include pre-treatment behaviour such as usage in the weeks before adoption, which is a strong proxy for motivation. I must avoid post-treatment variables, because adjusting for things the feature itself changes removes part of the real effect or adds bias. Next I check overlap: if many adopters have propensities near one, there are no comparable non-adopters, so I trim and state the population the estimate applies to. After matching or weighting I check balance with standardised mean differences below about 0.1. Finally I run a sensitivity analysis to see how strong a hidden confounder would need to be to remove the effect, and use a negative control outcome the feature should not affect.

You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?

What the interviewer wants to hear

Model answer

More hard Data Scientist questions