hard20 pts · first answer countsab-testingexperiment-design

Lift that fades after week one

A redesigned home page shows a 12% lift in clicks in week one, 5% in week two and 1% in week three. What is the most likely explanation, and how would you check it?

A.The test lost power over time; add more usersB.A novelty effect: returning users click the new design because it is new; compare new and returning users and look at the effect by days since first exposureC.Seasonality made week one special; nothing else to checkD.The randomisation failed in week one only

← Uneven groups in a 50/50 test Randomised by user, analysed by session →