Skip to content
RED SAGSkills
mediumDigital Marketing Specialist interview question

Walk me through how you plan and run an A/B test.

What the interviewer wants to hear

  • Start with a data-backed hypothesis
  • Choose one primary metric and guardrails
  • Calculate sample size and duration in advance
  • Randomise and run for full business cycles without peeking
  • Analyse with significance and confidence intervals
  • Document and share learnings

Model answer

I start with a hypothesis based on data, for example: 'Heatmaps show few people scroll to the pricing table, so moving it higher will increase trial sign-ups.' I choose one primary metric, such as trial sign-ups, and a few guardrail metrics like revenue per visitor. Before launch, I calculate the sample size needed to detect a realistic lift and turn it into a duration, usually at least one or two full weeks to cover weekday patterns. Traffic is split randomly and I avoid stopping early just because the result looks good, because peeking inflates false positives. After the test, I check significance and the confidence interval, and look at key segments such as mobile and desktop. Finally, I document the result, including failed tests, in a shared log so the team learns and does not repeat ideas.

Can you answer it under time pressure?

Take a timed medium round and get scored on every answer.

Start mock interview

More medium Digital Marketing Specialist questions