hard20 pts · first answer countsab-testingstatistics

Testing five variants at once

You test five variants against a control, each at 95% significance, and one variant 'wins'. What is the main concern?

A.With five comparisons, the chance of at least one false winner is about 23%, so adjust for multiple comparisons or confirm with a retestB.None, because 95% significance protects every comparisonC.The test is invalid because only two variants are allowedD.The winning variant is certain to be 5 times better

← How big must the test be?Measuring YouTube's real impact →