Data Scientist
A/B testing and causal inference questions
77 questions on A/B testing and causal inference for data scientists, easiest first.
Easy 17
- Meaning of a p-valueeasy
- Ice cream and drowningeasy
- Type I erroreasy
- Unit of randomisation for a new menueasy
- Why run an A/A testeasy
- What MDE meanseasy
- The counterfactualeasy
- Why random assignment workseasy
- Same user, same varianteasy
- When a bandit beats an A/B testeasy
- Keeping a long-term holdouteasy
- Running a test for 10 dayseasy
- Comparing beta users with everyone elseeasy
- Ramping traffic from 1% upwardseasy
- Writing the null hypothesiseasy
- A real effect that was missedeasy
- z-test or t-testeasy
Medium 34
- Sample size for a smaller effectmedium
- Reading a 95% confidence intervalmedium
- Testing 20 metrics at oncemedium
- Overall rate flips when splitmedium
- Studying only active usersmedium
- Users needed to detect a 1 point liftmedium
- What statistical power meansmedium
- Comparing means with unequal variancesmedium
- Testing two categorical variablesmedium
- Bonferroni threshold for ten testsmedium
- A lift that fades after week onemedium
- PM wants to ship an unclear winmedium
- Analysing only users who reached checkoutmedium
- From triggered lift to overall liftmedium
- One huge order swings the resultmedium
- Users who never opened the featuremedium
- Computing a difference-in-differencesmedium
- What propensity matching doesmedium
- Which variable to control formedium
- Agreeing on one decision metricmedium
- Two tests running at the same timemedium
- Probability to beat controlmedium
- A rule change as an experimentmedium
- Testing a billboard campaignmedium
- Groups differed before the test beganmedium
- Randomising by device for logged-in usersmedium
- Only 40 stores to test onmedium
- Choosing a one-sided testmedium
- Before and after on the same usersmedium
- Are orders spread evenly over weekdaysmedium
- Comparing average baskets in four citiesmedium
- Small, very skewed samplesmedium
- Significant but not importantmedium
- A coefficient with p = 0.30medium
Hard 26
- Checking results every dayhard
- Coaching the worst storeshard
- Unexpected group sizeshard
- Using pre-experiment datahard
- Difference-in-differences assumptionhard
- Feature users retain betterhard
- Click-through rate standard errorshard
- Propensity scores near 0 and 1hard
- Key assumption of an instrumenthard
- A link that only appears in a subsethard
- Direction of omitted variable biashard
- Treated drivers take control rideshard
- Benjamini-Hochberg discoverieshard
- Big effect in one segmenthard
- Testing a group-chat featurehard
- Testing a new dispatch algorithmhard
- Looking at results weekly, done righthard
- Effect on those who actually used ithard
- Free delivery above ₹499hard
- A launch in only one cityhard
- Adjusting for a variable on the causal pathhard
- Is 10% versus 11% significanthard
- Comparing two search rankers quicklyhard
- Who should get the couponhard
- Power of a 90/10 splithard
- How a permutation test workshard