Your A/B test finished and the result is not statistically significant. What do you do next?
What the interviewer wants to hear
- Check test validity: sample ratio, tracking, duration
- Look at confidence interval and effect size
- Do not hunt through segments for a win
- Decide based on cost, risk and strategy
- Learn: revisit hypothesis, consider qualitative research
Model answer
First I check the test was valid: the traffic split matched the plan, tracking worked and it ran for full weeks with the planned sample size. Then I look at the confidence interval instead of just the p-value. If the interval is narrow around zero, the change probably has little effect. If it is wide, the test was underpowered and we simply do not know. I avoid slicing results into many segments to find a winner, since that produces false positives; any segment idea becomes a new hypothesis. The decision then depends on context: if the change is cheap, low risk and strategically useful, we might ship it anyway; if it adds complexity, we keep the old version. Finally I ask what we learned, possibly run quick user interviews, and design a bolder next experiment.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Product Manager questions
- How would you define success metrics for a new feature?
- How do you say no to a stakeholder request?
- How do you run customer discovery for a new problem area?
- Tell me about a feature that did not perform as expected. What did you do?
- How do you write a product spec or PRD that engineers find useful?
- Leadership wants to change your product's pricing. How would you approach it?
- Walk me through how you plan a major product launch.
- How would you set OKRs for your product team for next quarter?
- How would you tell whether a new product has product-market fit?