Daily active users dropped 8% overnight. How do you investigate?
What the interviewer wants to hear
- Confirm the drop is real: data pipeline, tracking, definitions
- Check external factors: holidays, outages, competitor events
- Segment by platform, version, region, channel
- Correlate with releases and experiments
- Communicate findings and next steps quickly
Model answer
First I check whether the drop is real. I look for pipeline delays, missing partitions, changes in event tracking or a new metric definition, and compare with other sources like server logs or backend orders. If the data is fine, I check the calendar and outside events: a festival, a public holiday, a payment or app store outage. Then I segment: platform, app version, country, acquisition channel, new versus returning users. A sharp drop limited to one segment usually points to the cause, like Android users on the latest release, which suggests a bug in that version, or one region, suggesting an outage or network issue. I check release notes, feature flags and experiments launched that day. I share a short update early with what we know and do not know, then the root cause and fix once confirmed.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Data Scientist questions
- An A/B test shows a significant 3% increase in conversion but a significant 2% drop in average order value. What do you do?
- Leadership wants to know the effect of a loyalty programme that launched for everyone last year, with no experiment. How do you estimate it?
- Build a daily demand forecast for 5,000 products across 200 stores. How would you approach it?
- Why are A/B tests tricky in a two-sided marketplace like a ride-hailing app, and how would you handle it?
- Marketing wants to send a discount only to customers it will actually persuade. How would you build and validate that targeting model?
- A model that looked strong offline performs much worse a month after launch. How do you investigate?
- When would you use survival analysis instead of a binary churn classifier, and how would you set it up?
- You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?
- The business wants to know how demand responds to price for thousands of products. How would you estimate price elasticity?