hard20 pts · first answer countsstatisticsab-testing

Overall rate goes the other way

Version B has a higher conversion rate than A on both mobile and desktop, but A has a higher conversion rate overall. What explains this?

A.The test must have a bug, because this is mathematically impossibleB.B received a much larger share of low-converting mobile traffic, so mixing segments reverses the overall result (Simpson's paradox)C.The p-value is too highD.Desktop users are counted twice

Stopping a test early →