medium10 pts · first answer countsab-testingdata-informed-design

Reading an early A/B test result

One day into an A/B test, with 200 users, the new design shows 2% higher sign-ups. The PM wants to ship it now. What do you recommend?

A.Ship it today, since it is already winningB.Run the test until the planned sample size and duration, then check statistical significance and guardrail metricsC.Stop the test, because 2% is too small to matterD.Ask the team which version they like better

← Writing a helpful form error Jobs to be Done framing →