The product team wants to test a new checkout button. How would you design the A/B test?

What the interviewer wants to hear

• Clear hypothesis and one primary metric
• Guardrail metrics
• Randomise at the right unit such as user
• Calculate sample size and duration in advance
• Run full weeks and avoid peeking
• Check sample ratio mismatch before reading results

Model answer

I start with a hypothesis: the new button will increase checkout completion. The primary metric is checkout conversion per user who reaches the cart, and guardrails include average order value, payment errors and page load time. I randomise by user, not by session, so a person always sees the same version. Using the baseline conversion, the smallest lift worth detecting, 5% significance and 80% power, I calculate the sample size and the number of days needed, rounding up to full weeks to cover weekday and weekend behaviour. I agree on this plan before launch and do not stop early when results look good. When the test ends, I first check that the traffic split matches the plan, then look at the effect size with a confidence interval, the guardrails, and key segments before making a recommendation.

More medium Data Analyst questions

• Daily orders dropped 20% yesterday. Walk me through how you would investigate.
• What are SQL window functions and when have you used them?
• How would you choose the key metrics for a food delivery app?
• When would you use Excel, SQL or Python for an analysis task?
• What is cohort analysis and when would you use it instead of looking at overall totals?
• Three managers send you urgent data requests on the same morning. How do you decide what to do first?
• Checkout conversion has fallen. How would you analyse the funnel to find where users are dropping off?
• How do you decide what to do with outliers in your data?
• How would you design the data model for a new Power BI sales report?