Digital Marketing Specialist
A/B testing and causal inference questions
50 questions on A/B testing and causal inference for digital marketing specialists, easiest first.
Easy 13
- Writing a test hypothesiseasy
- What an A/B test iseasy
- Why visitors are assigned at randomeasy
- Reading "significant at 95%"easy
- Why a test needs enough visitorseasy
- Running tests for full weekseasy
- Running an A/A testeasy
- Picking the main metric in advanceeasy
- Calculating relative lifteasy
- When to use a split URL testeasy
- What test flicker iseasy
- What a guardrail metric iseasy
- Changing many things in one varianteasy
Medium 14
- Design a clean A/B testmedium
- What 95% significance meansmedium
- Checking results every daymedium
- One of 20 metrics is significantmedium
- Bandit or A/B testmedium
- Server-side or client-side testsmedium
- A/B tests and search enginesmedium
- Why keep a holdback after launchmedium
- A 60% lift on day twomedium
- A win with a broken guardrailmedium
- How long will the test take?medium
- A 90/10 traffic splitmedium
- Choosing a test's success metricmedium
- What significance tells youmedium
Hard 23
- Stopping an A/B test too earlyhard
- How big must the test be?hard
- Testing five variants at oncehard
- A shrinking test lifthard
- Judging a redesign by before and afterhard
- Testing a low-traffic pagehard
- What CUPED doeshard
- Randomise by user, analyse by sessionhard
- Two tests on the same pagehard
- Diluted effect from untouched usershard
- Peeking safely during a testhard
- Keeping conflicting tests aparthard
- Measuring a personalisation enginehard
- Is the exit-intent discount paying off?hard
- A chatbot that opens by itselfhard
- Is a 3% vs 2.5% click gap real?hard
- Sample size for a click-rate testhard
- Testing a trial expiry emailhard
- Proving a channel is incrementalhard
- Designing a geo-lift testhard
- Stopping a test earlyhard
- Reading a holdout resulthard
- A clean creative testhard