Walk through how you would design an A/B test for a new checkout page.

What the interviewer wants to hear

• Clear hypothesis and primary metric
• Guardrail metrics
• Randomisation unit and power analysis for sample size and duration
• Checks: sample ratio mismatch, novelty effect, no peeking
• Decision rule agreed before launch

Model answer

I start with a hypothesis: the new checkout reduces friction, so the purchase conversion rate per visitor will go up. That is my primary metric. I add guardrails like average order value, payment errors, page load time and refund rate, so a "win" does not hide damage. I randomise by user, so people see a consistent page across sessions. Then I run a power analysis using the baseline conversion, its variance and the smallest lift worth detecting to decide sample size, and I run for at least one or two full weeks to cover weekday patterns. During the test I check for sample ratio mismatch and logging problems, but I do not stop early just because the p-value looks good. Before launch, the team agrees on the decision rule, for example ship if conversion improves significantly and no guardrail worsens.

More medium Data Scientist questions

• How would you choose a success metric for a new "recommended for you" section in a shopping app?
• You have 500 candidate features for a churn model. How do you decide which ones to keep?
• A business team does not trust your credit risk model. How do you explain its predictions?
• The CEO asks: "Why are customers churning?" How do you approach this?
• You are building a model where only 0.5% of examples are positive. How do you approach training and evaluation?
• How would you validate a demand forecasting model before the business relies on it?
• What is data leakage, how do you detect it, and how do you prevent it in a modelling project?
• How do you decide between a regularised linear model and a gradient boosted tree model for a tabular problem?
• Walk me through how you would decide the sample size and duration of an A/B test.