medium10 pts · first answer countsab-testing

A lift that fades after week one

A redesigned home page shows a 6% engagement lift in week one, 3% in week two and 1% in week three. What is the most likely explanation, and what should you do?

A.A novelty effect; run longer and look at the effect over time or for new users onlyB.Sample ratio mismatch; restart the testC.The test is now overpowered; stop immediatelyD.Seasonality; the lift is real and stable

← Lift of the top decile PM wants to ship an unclear win →