An A/B test shows a significant 3% increase in conversion but a significant 2% drop in average order value. What do you do?

What the interviewer wants to hear

• Check validity first: SRM, bugs, novelty, duration
• Look at the overall metric that matters, such as revenue per visitor
• Segment to understand the mechanism
• Consider long-term effects like margin and retention
• Make a recommendation with trade-offs

Model answer

First I check the test is valid: no sample ratio mismatch, no logging changes, enough duration to pass novelty effects, and the metrics were planned in advance. Then I look at what actually matters for the business, usually revenue or profit per visitor, which combines conversion and order value. A 3% rise in conversion and a 2% fall in order value is roughly a 1% revenue gain, but I would compute it directly with a confidence interval. Next I segment to understand the mechanism: maybe the new design brings in more small first-time purchases, which could be good for long-term retention, or maybe it pushes cheaper items with lower margin. I would check margin, return rate and repeat purchase rate. My recommendation would explain the trade-off clearly and might suggest a follow-up test to keep the conversion gain while protecting order value.

More hard Data Scientist questions

• Leadership wants to know the effect of a loyalty programme that launched for everyone last year, with no experiment. How do you estimate it?
• Build a daily demand forecast for 5,000 products across 200 stores. How would you approach it?
• Why are A/B tests tricky in a two-sided marketplace like a ride-hailing app, and how would you handle it?
• Daily active users dropped 8% overnight. How do you investigate?
• Marketing wants to send a discount only to customers it will actually persuade. How would you build and validate that targeting model?
• A model that looked strong offline performs much worse a month after launch. How do you investigate?
• When would you use survival analysis instead of a binary churn classifier, and how would you set it up?
• You must estimate the effect of a feature that users chose to adopt, using propensity score methods. What can go wrong and how do you guard against it?
• The business wants to know how demand responds to price for thousands of products. How would you estimate price elasticity?