Why are A/B tests tricky in a two-sided marketplace like a ride-hailing app, and how would you handle it?
What the interviewer wants to hear
- Interference: treatment and control share the same supply
- User-level randomisation can be biased
- Switchback tests by time and region, or cluster randomisation
- Trade-off between bias and statistical power
- Measure both sides of the market
Model answer
In a marketplace, riders in treatment and control compete for the same drivers. If a new pricing feature makes treatment riders book more, they take drivers away from control riders, so control gets worse and the measured lift is exaggerated. This breaks the assumption that one user outcome does not depend on another group. To reduce interference, I randomise at a level where spillover is small. Switchback experiments alternate treatment and control for a whole city or zone over time windows, like 30 minutes, so the whole market is in one state at a time. Cluster randomisation by city or region also works. The trade-off is fewer independent units, so less power and wider confidence intervals; I need longer tests and careful variance estimation. I would also measure both sides: rider conversion and wait times, and driver earnings and utilisation.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Data Scientist questions
- An A/B test shows a significant 3% increase in conversion but a significant 2% drop in average order value. What do you do?
- Leadership wants to know the effect of a loyalty programme that launched for everyone last year, with no experiment. How do you estimate it?
- Build a daily demand forecast for 5,000 products across 200 stores. How would you approach it?
- Daily active users dropped 8% overnight. How do you investigate?
- Marketing wants to send a discount only to customers it will actually persuade. How would you build and validate that targeting model?
- A model that looked strong offline performs much worse a month after launch. How do you investigate?
- When would you use survival analysis instead of a binary churn classifier, and how would you set it up?
- You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?
- The business wants to know how demand responds to price for thousands of products. How would you estimate price elasticity?