How would you choose a success metric for a new "recommended for you" section in a shopping app?
What the interviewer wants to hear
- Tie the metric to user value and business goals
- Avoid vanity metrics like raw clicks alone
- Primary metric plus counter or guardrail metrics
- Metric must be sensitive and measurable in a reasonable time
- Consider long-term effects such as cannibalisation
Model answer
I start with the goal: help users find products they want, which should increase purchases. Clicks on the section are easy to measure but can be a vanity metric, since eye-catching items get clicks without sales. So I would choose something closer to value, like purchases or revenue per user that come from the section, or better, overall purchases per user, because the section could just steal purchases from search. That overall metric captures cannibalisation. I would add guardrails: return rate, page load time, and diversity of items shown. The metric must also be sensitive enough to move within a few weeks of testing; if overall revenue is too noisy, I might use a proxy like add-to-cart rate, after checking that it predicts revenue. Longer term, I would watch repeat visits.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Data Scientist questions
- Walk through how you would design an A/B test for a new checkout page.
- You have 500 candidate features for a churn model. How do you decide which ones to keep?
- A business team does not trust your credit risk model. How do you explain its predictions?
- The CEO asks: "Why are customers churning?" How do you approach this?
- You are building a model where only 0.5% of examples are positive. How do you approach training and evaluation?
- How would you validate a demand forecasting model before the business relies on it?
- What is data leakage, how do you detect it, and how do you prevent it in a modelling project?
- How do you decide between a regularised linear model and a gradient boosted tree model for a tabular problem?
- Walk me through how you would decide the sample size and duration of an A/B test.