A/B Testing
36 practice items · needed by 7 roles
Easy 1Medium 14Hard 21
Practice questions
- Better offline metric, worse A/B testhard
- Sample size for a smaller effectmedium
- Reading a 95% confidence intervalmedium
- Testing 20 metrics at oncemedium
- Users needed to detect a 1 point liftmedium
- A lift that fades after week onemedium
- PM wants to ship an unclear winmedium
- Checking results every dayhard
- Unexpected group sizeshard
- Using pre-experiment datahard
- Click-through rate standard errorshard
- Treated drivers take control rideshard
- Big effect in one segmenthard
- What a p-value meansmedium
- Reading a 95% confidence intervalmedium
- False positive in an A/B testmedium
- Relative lift of a variantmedium
- Overall rate goes the other wayhard
- Stopping a test earlyhard
- Detecting a small lifthard
- Uneven groups in a 50/50 testhard
- Lift that fades after week onehard
- Randomised by user, analysed by sessionhard
- Reading an early A/B test resultmedium
- What caused the lifthard
- An early spike after launchhard
- Wins on mobile, loses on desktophard
- The control groupeasy
- Design a clean A/B testmedium
- What 95% significance meansmedium
- Stopping an A/B test too earlyhard
- How big must the test be?hard
- Testing five variants at oncehard
- A shrinking test lifthard
- Email subject line testmedium
- Calling a headline test too earlyhard