medium10 pts · first answer countsmultiple-testingab-testing

Testing 20 metrics at once

An experiment shows a significant change (p < 0.05) in 1 out of 20 secondary metrics, while the primary metric is flat. What is the right reading?

A.The change clearly improved that metric; ship itB.All 20 metrics must be wrongC.Increase alpha to 0.10 so more metrics become significantD.With 20 tests at 0.05, about one false positive is expected; apply a correction such as Bonferroni or Benjamini-Hochberg and treat it as a hypothesis to retest

← Reading a 95% confidence interval Latest order per customer →