An A/B test shows a significant lift in conversion but a drop in average order value, and revenue per user is flat. What do you recommend?

What the interviewer wants to hear

• Decide based on the metric closest to business value, such as revenue or profit per user
• Check data quality and sample ratio mismatch first
• Look at the confidence intervals, not just significance
• Segment to understand who changed behaviour
• Consider long-term effects like retention
• Possibly iterate or run a follow-up test

Model answer

First I verify the test is valid: traffic split matches the plan, no tracking bugs, and enough runtime. Then I go back to the goal. Conversion and order value are both inputs; the business usually cares about revenue or margin per user. If revenue per user is flat with a confidence interval around zero, the change is not clearly winning. I segment to see what happened: maybe more small first-time buyers converted, which lowers average order value but could be valuable if they come back. So I would check repeat purchase or retention for the new converters, and margin, since small orders may cost more to deliver. My recommendation might be: do not roll out yet, but if new-customer acquisition is the strategic goal, run a longer test or a follow-up that measures 30-day value. I present this trade-off clearly instead of just reporting the significant metric.

More hard Data Analyst questions

• Finance and Marketing report different revenue numbers for the same month. How do you resolve it?
• A feature was launched to all users at once, so there is no control group. How would you estimate its impact?
• Your team gets 50 ad-hoc data requests a week. How would you reduce that load without hurting the business?
• An experiment report checks 20 metrics across 10 segments and highlights the 3 that are significant at p < 0.05. How do you judge it?
• The CFO asks you to forecast next quarter revenue. How do you approach it, and how do you communicate uncertainty?
• A senior leader asks you to "find data that proves" their new campaign worked. Your analysis suggests it did not. What do you do?
• You run an A/B test on a ride-hailing or delivery marketplace where riders and drivers share the same supply. Why might a normal user-level test give a wrong answer, and what would you do?
• Marketing wants to segment customers for targeted campaigns. How would you build the segmentation, and how would you know it is useful?
• Your executive dashboard was wrong twice last quarter and leaders have stopped trusting it. How do you rebuild trust?