An experiment report checks 20 metrics across 10 segments and highlights the 3 that are significant at p < 0.05. How do you judge it?

What the interviewer wants to hear

• Explain multiple comparisons and expected false positives
• Primary metric should be decided before the test
• Use corrections such as Bonferroni or false discovery rate control
• Treat segment findings as hypotheses to confirm in a new test
• Look at effect sizes and whether results make sense

Model answer

With 200 metric and segment combinations, at a 5% threshold you would expect about 10 to look significant by pure chance, so 3 significant results is actually not evidence of anything. This is the multiple comparisons problem. The correct approach is to fix one primary metric and a few guardrails before the test and judge the decision mainly on those. For secondary metrics, apply a correction such as Bonferroni, which is strict, or Benjamini-Hochberg to control the false discovery rate. Segment results that were not planned should be treated as ideas, not conclusions, and confirmed in a new test designed for that segment. I also look at whether the effect size is meaningful and whether there is a believable reason for it. I would explain this to the team kindly, because highlighting lucky results can lead to bad product decisions.

More hard Data Analyst questions

• An A/B test shows a significant lift in conversion but a drop in average order value, and revenue per user is flat. What do you recommend?
• Finance and Marketing report different revenue numbers for the same month. How do you resolve it?
• A feature was launched to all users at once, so there is no control group. How would you estimate its impact?
• Your team gets 50 ad-hoc data requests a week. How would you reduce that load without hurting the business?
• The CFO asks you to forecast next quarter revenue. How do you approach it, and how do you communicate uncertainty?
• A senior leader asks you to "find data that proves" their new campaign worked. Your analysis suggests it did not. What do you do?
• You run an A/B test on a ride-hailing or delivery marketplace where riders and drivers share the same supply. Why might a normal user-level test give a wrong answer, and what would you do?
• Marketing wants to segment customers for targeted campaigns. How would you build the segmentation, and how would you know it is useful?
• Your executive dashboard was wrong twice last quarter and leaders have stopped trusting it. How do you rebuild trust?