hard20 pts · first answer countsab-testingdata-quality

Unexpected group sizes

An experiment was configured as a 50/50 split. After 1 million users, control has 512,000 and treatment has 488,000. What should you do?

A.Nothing; small differences are normal at this scaleB.Reweight the groups to 50/50 and report the resultsC.Drop the extra control users at randomD.Treat it as a sample ratio mismatch: a chi-square test shows this is very unlikely by chance, so investigate assignment or logging bugs before trusting any results

← Coaching the worst stores Using pre-experiment data →