hard20 pts · first answer countsexperimentationstatistics

A positive but uncertain result

An A/B test shows a +2% lift in conversion with a 95% confidence interval from -1% to +5%. The team wants to announce a win. What is the correct reading?

A.The feature definitely increases conversion by 2%B.The feature definitely hurts conversionC.The result proves there is no effectD.The result is inconclusive: the true effect could be negative, zero or positive; decide based on cost, risk and whether a longer test is worth it

← Reversible and irreversible decisions How long will the test take →