medium10 pts · first answer countsexperimentationstatistics

Checking results every day

A PM checks a two-week A/B test every morning and stops it on day 3 when it first shows p < 0.05. What is the problem?

A.Repeatedly peeking and stopping at the first significant result greatly increases false positives; decide the sample size up front or use a sequential testing methodB.Day 3 is a WednesdayC.There is no problem; significance is significanceD.They should have stopped on day 1

← Guardrail metrics An uneven 50/50 split →