You run an A/B test on a ride-hailing or delivery marketplace where riders and drivers share the same supply. Why might a normal user-level test give a wrong answer, and what would you do?

Question

RED SAG · Accepted Answer

Standard A/B tests assume one user's treatment does not affect another user, but in a marketplace it does. If a variant gives some riders a discount, they book more and take drivers away from control riders. Control looks worse and the variant looks better, so the measured lift is exaggerated, and the real effect at full launch may be near zero. To reduce this I would randomise at a level where the shared supply is contained. A switchback test alternates the whole city between control and treatment in time blocks, such as 30 minutes. A geo test randomises whole zones or cities. These designs have fewer independent units, so power is lower and I need longer tests and cluster-aware analysis. I would also track both sides of the market, like driver earnings and wait times, not just rider conversions.

You run an A/B test on a ride-hailing or delivery marketplace where riders and drivers share the same supply. Why might a normal user-level test give a wrong answer, and what would you do?

What the interviewer wants to hear

Model answer

More hard Data Analyst questions