hard20 pts · first answer countsab-testingmultiple-testing

Big effect in one segment

An A/B test shows no overall effect. After slicing by 30 segments, one segment (Android users in Brazil) shows a large significant lift with p = 0.02. What should you do?

A.Ship the change to that segment immediatelyB.Report it as the main finding of the testC.Treat it as a hypothesis likely to be a false positive, and confirm it in a new pre-registered testD.Drop the other segments and re-run the analysis

← Optimising time on site Take the hard mock interview →