A model that looked strong offline performs much worse a month after launch. How do you investigate?

What the interviewer wants to hear

• Confirm the drop is real and measured correctly, given label delay
• Check training-serving skew and pipeline bugs first
• Compare feature and score distributions with training (data drift)
• Check whether the relationship has changed (concept drift)
• Look for leakage in the offline evaluation and feedback loops

Model answer

First I confirm the drop is real: are labels complete yet, is the metric computed the same way as offline, and did the population change, for example a new market? Then I look for training-serving skew, which is the most common cause. I log the features the model actually received and compare them with what the offline pipeline would compute for the same entities; differences in units, time zones, default values or missing joins show up quickly. Next I compare feature and score distributions with training data using PSI or simple histograms to find data drift. If inputs look the same but performance fell, the relationship may have changed, which is concept drift, perhaps from a competitor or policy change. I also revisit the offline evaluation for leakage, since a too-good backtest often explains a disappointing launch, and check for feedback loops where the model changes the data it later learns from.

More hard Data Scientist questions

• An A/B test shows a significant 3% increase in conversion but a significant 2% drop in average order value. What do you do?
• Leadership wants to know the effect of a loyalty programme that launched for everyone last year, with no experiment. How do you estimate it?
• Build a daily demand forecast for 5,000 products across 200 stores. How would you approach it?
• Why are A/B tests tricky in a two-sided marketplace like a ride-hailing app, and how would you handle it?
• Daily active users dropped 8% overnight. How do you investigate?
• Marketing wants to send a discount only to customers it will actually persuade. How would you build and validate that targeting model?
• When would you use survival analysis instead of a binary churn classifier, and how would you set it up?
• You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?
• The business wants to know how demand responds to price for thousands of products. How would you estimate price elasticity?