Walk me through how you would decide the sample size and duration of an A/B test.
What the interviewer wants to hear
- Choose the primary metric and its baseline and variance
- Agree a minimum detectable effect that matters to the business
- Set alpha and power, then compute users per group
- Convert users to days using traffic, and cover full weekly cycles
- Commit to the duration in advance to avoid peeking
Model answer
I start with the primary metric and get its baseline value and variance from historical data. Then I agree a minimum detectable effect with the product team, meaning the smallest change worth acting on, not the effect we hope for. With alpha usually 0.05 and power 80%, I compute the sample size per group; for a conversion rate this is roughly 16 times p(1-p) divided by the effect squared. I divide that by the daily eligible traffic to get the number of days, then round up to full weeks so weekday and weekend behaviour are both included. If the duration is too long, I look at variance reduction such as CUPED, a more sensitive metric, or a larger effect size. Finally I write the plan down before launch, including the stopping date, so we do not peek and stop early on a lucky result.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Data Scientist questions
- Walk through how you would design an A/B test for a new checkout page.
- How would you choose a success metric for a new "recommended for you" section in a shopping app?
- You have 500 candidate features for a churn model. How do you decide which ones to keep?
- A business team does not trust your credit risk model. How do you explain its predictions?
- The CEO asks: "Why are customers churning?" How do you approach this?
- You are building a model where only 0.5% of examples are positive. How do you approach training and evaluation?
- How would you validate a demand forecasting model before the business relies on it?
- What is data leakage, how do you detect it, and how do you prevent it in a modelling project?
- How do you decide between a regularised linear model and a gradient boosted tree model for a tabular problem?