Leadership wants to know the effect of a loyalty programme that launched for everyone last year, with no experiment. How do you estimate it?

What the interviewer wants to hear

• State the causal question and assumptions clearly
• Methods: difference-in-differences, synthetic control, matching, regression discontinuity, interrupted time series
• Choose based on data available (control groups, timing, eligibility rules)
• Robustness checks and placebo tests
• Communicate uncertainty and limitations

Model answer

I start by defining the question exactly: the effect of joining, or of the programme existing, on spend over what time frame. Then I look for natural comparisons. If the programme rolled out in different cities at different times, difference-in-differences comparing early and late cities works, after checking parallel pre-trends. If there was an eligibility rule, like spending above a threshold, regression discontinuity compares customers just above and below. If all customers got it at once, synthetic control or interrupted time series with a forecast of the counterfactual can help, using similar markets or products as controls. Comparing members to non-members needs matching on pre-launch behaviour, but self-selection bias remains. I run placebo tests, such as fake launch dates, and try several methods. I report a range, not one number, and state the assumptions clearly.

More hard Data Scientist questions

• An A/B test shows a significant 3% increase in conversion but a significant 2% drop in average order value. What do you do?
• Build a daily demand forecast for 5,000 products across 200 stores. How would you approach it?
• Why are A/B tests tricky in a two-sided marketplace like a ride-hailing app, and how would you handle it?
• Daily active users dropped 8% overnight. How do you investigate?
• Marketing wants to send a discount only to customers it will actually persuade. How would you build and validate that targeting model?
• A model that looked strong offline performs much worse a month after launch. How do you investigate?
• When would you use survival analysis instead of a binary churn classifier, and how would you set it up?
• You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?
• The business wants to know how demand responds to price for thousands of products. How would you estimate price elasticity?