hard20 pts · first answer countsexperimentationstatistics
Twenty metrics, one winner
An A/B test with no real effect tracks 20 independent metrics, each tested at p < 0.05. Roughly what is the chance that at least one shows a "significant" result by luck?