hard20 pts · first answer countsab-testingstatistics

Randomised by user, analysed by session

Users were randomised into control and variant, but the analyst ran a standard test treating each session as an independent observation. What is the main risk?

A.Sessions from the same user are correlated, so the standard error is too small and the p-value looks more significant than it isB.The test will have too little dataC.Sessions cannot be counted in SQLD.There is no risk because sessions are a finer level

← Lift that fades after week one Halving the margin of error →