A/B Testing interview questions and answers

Answer: A. Rejecting the null hypothesis when it is actually true (a false positive)

A Type I error is a false positive: concluding there is an effect when there is none. Its rate is controlled by the significance level alpha, often 0.05. A Type II error is a false negative, and its rate relates to the power of the test.

Answer: B. To check the system's false positive rate

With identical experiences, any 'significant' result is a false positive, so about 5% of A/A tests should be significant at α = 0.05. Many more points to broken randomisation, logging bugs or wrong variance formulas. It is a cheap way to trust the system.

Answer: D. What the same customer would have done without the coupon

The causal effect for one person is the difference between what happened and what would have happened under the other choice. We never see both, which is the core problem of causal inference. Randomised control groups estimate the average counterfactual.

Answer: A. It balances other factors, even unseen ones

Random assignment makes treatment independent of user traits like income, intent or device. So on average the groups differ only in treatment, and chance differences are what the p-value and interval account for. Observational comparisons lack this guarantee.

Answer: B. Each user keeps the same variant every visit

A deterministic hash gives each user a stable bucket without storing assignments, so returning users keep their variant. Including the experiment ID gives independent splits across experiments. A fresh random draw per visit would mix experiences.

Answer: D. A long-term holdout without the changes

A small holdout, say 2–5% of users kept on the old experience for months, measures the combined effect of all launches against a true baseline. Summed lifts are usually inflated by winner's curse and interactions. Holdouts also reveal long-term effects such as fatigue.

Answer: A. An A/B test measures business impact between versions; a canary checks a release is safe before full rollout

A canary is a safety step: send a small share of traffic to the new version and watch errors, latency and basic quality before expanding. An A/B test splits traffic for long enough to compare business metrics with statistical confidence. Many teams canary first, then run an A/B test.

Answer: B. The smallest product that lets you test a key assumption with real users and learn quickly

An MVP is about learning with minimum effort. It must still be valuable and usable enough to test the main assumption, such as 'Will small shops pay for this?' It is not a buggy or full-featured product.

Answer: C. Users who keep the current experience, so you can compare results against them

The control shows what would have happened without the change. Because users are randomly assigned, differences between control and treatment can be attributed to the change, not to outside factors like seasonality.

Answer: D. We believe that showing a setup checklist to new users will increase 7-day activation from 30% to 35%, because interviews show people do not know the next step

A good hypothesis states the change, the expected measurable effect and the reasoning. This makes it testable and tells you clearly whether you were right, so the team learns either way.

Answer: C. The one most likely wrong and most damaging if so

Testing the riskiest, most important assumption first avoids spending months on something that a single fact could kill. Cheap tests are good, but only if they target what matters.

Answer: A. "Users miss delivery dates, so showing them by the CTA will lift add-to-cart"

A good hypothesis links evidence (what you observed), a specific change, and an expected, measurable result. That makes the test useful even if it loses, because you learn whether the underlying idea was right.

Answer: B. With so few visitors, chance alone easily creates such gaps

At 200 visitors, each group has around 100 people and only 3 to 5 conversions. Gaps like this happen by luck all the time. Plan the sample size before starting, and only judge the test once it is reached.

Answer: B. Behaviour differs by weekday; full weeks balance it

Visitors behave differently on weekdays and weekends, and around paydays. Ending a test mid-week can over-represent some days. Covering full weekly cycles gives a fairer picture of typical behaviour.

Answer: C. So you cannot pick whichever metric happens to look good

If you choose the metric after seeing results, some metric will usually look like a winner by chance. Fixing the primary metric, and the guardrails, in advance keeps the decision honest.

Answer: C. A split URL test that sends visitors to either page

Split URL (redirect) tests send each visitor to one of two separate pages. They suit large redesigns that would be hard to build as on-page changes. Use temporary redirects and canonical tags so search engines are not confused.

Answer: B. Flicker (flash of original content)

Flicker happens when the page renders before the testing script applies changes. It looks broken and can bias results. Faster script loading, anti-flicker snippets or server-side testing reduce it.

Answer: A. They are guardrails that catch harm the main metric misses

Guardrail metrics protect against hidden damage, such as more errors, refunds or slower pages. A variant that wins on conversion but breaks a guardrail may not be safe to ship.

Answer: A. You cannot tell which change caused the win

Bundled changes can be a good way to get a big, detectable lift, but they do not show which part worked. If learning matters, test key changes separately afterwards, or use a multivariate design when traffic allows.

Answer: C. You cannot tell which change caused any difference

Changing several things at once mixes up their effects. To learn about the headline, keep the image and offer the same and change only the headline. Multivariate tests can study several changes but need far more traffic.

Answer: A. It becomes about 4 times larger

Required sample size scales with one over the square of the minimum detectable effect. Halving the effect you want to detect means about four times the sample. This is why small lifts need long tests or variance reduction techniques.

Answer: B. About 15,000

Using n = (1.96 + 0.84)^2 x (p1(1-p1) + p2(1-p2)) / d^2 = 7.84 x (0.09 + 0.099) / 0.0001 gives about 14,800 per group. A handy rule of thumb is 16 x p(1-p) / d^2, which gives 14,400. Small absolute lifts on low base rates need large samples.

Answer: A. A novelty effect; run longer and look at the effect over time or for new users only

Existing users often click on anything new, which inflates early results before habits settle. Plotting the effect by week, or analysing only new users who never saw the old design, shows the long-run effect. Shipping on week-one numbers would overstate the benefit.

Answer: D. Analyse users who reached payment, in both groups

Users who never reach the changed step cannot be affected, so including them only adds noise. Triggered analysis keeps users in both groups who reached the point where the experience would differ, which keeps the comparison fair. Filtering on an outcome like conversion would bias it.

Answer: B. Income before joining, which affects both

A confounder affects both treatment and outcome; prior income makes people likelier to join and to spend. Adjusting for it removes that backdoor path. Variables affected by the treatment, such as later orders, should not be adjusted for, as that blocks part of the effect.

Answer: A. A 94% chance B's true rate is higher

A Bayesian posterior gives a direct probability that one variant's true rate exceeds the other's. It does not say how big the gain is, so also look at expected lift and risk of loss. It is not the same as 1 minus a p-value.

Answer: C. Run it in some cities; compare with matched ones

Geo experiments assign regions, not people, to treatment. Choosing cities that matched well on past sales, then comparing their trends, estimates the lift. Few units means low power, so careful matching and methods like synthetic control help.

Answer: D. Randomisation may be broken; investigate

With correct random assignment, pre-period metrics should be similar, apart from small chance gaps. A clear gap points to bucketing bugs, users carried over from an earlier test, or a rare chance imbalance. Methods like CUPED can adjust for small gaps, but large ones need a fix.

Answer: D. Assign by a hash of the user ID

Sticky assignment by user gives each person a consistent experience and makes metrics such as retention or satisfaction attributable to one model. A stable hash of the user ID is simple and needs no stored state.

Answer: C. 25%

Relative lift is (variant - control) / control = (5.0 - 4.0) / 4.0 = 25%. The absolute lift is 1 percentage point. Reports should state both, because a 25% lift on a tiny base rate may be worth far less than it sounds.

Answer: B. Run the test until the planned sample size and duration, then check statistical significance and guardrail metrics

Early results with small samples swing a lot, and stopping as soon as a test looks good inflates false positives. Decide the sample size and run time up front (often at least one full week to cover weekday patterns), then judge significance and guardrails.

Answer: D. They watch for harm, such as higher churn, more support tickets or lower satisfaction, so a win on the main metric does not hide damage elsewhere

A primary metric can be gamed by changes that hurt the wider business. Guardrails are agreed before launch with limits that stop or roll back the test. Common ones include latency, errors, refunds and unsubscribes.

Answer: A. Early, so the biggest doubt is tested first

Ordering work to test the biggest uncertainty first means a fatal problem is found early, before the rest is built. Later work can then be adjusted to what was learned.

Answer: A. Split traffic randomly between the old and new page, change only the headline, and run until a sample size decided in advance is reached

A clean A/B test changes one thing, splits traffic randomly at the same time and runs to a planned sample size. Changing many elements hides what caused the result, and week-by-week comparisons mix in seasonality and other changes. Colleague opinions are not customer behaviour.

Answer: D. For short-lived choices like a sale banner, where earning now matters

Bandits shift traffic toward better-performing variants as they learn, which earns more during short campaigns. The trade-off is less precise measurement of the effect, so classic A/B tests suit decisions you want to learn from.

Answer: B. Use 302 redirects and a canonical to the original URL

Google advises using temporary (302) redirects, rel=canonical pointing to the original, and not cloaking (showing Googlebot different content). Run tests only as long as needed, then remove test code.

Answer: A. Keep going to the planned sample; early swings are common

Early results swing widely and often shrink as data grows. Stopping early on a big number is a classic way to ship false or exaggerated wins. Stick to the plan unless a guardrail shows harm.

Answer: B. The test needs much more total traffic to reach the same power

Precision is limited by the smaller group. With only 10% in the variant, reaching the same power takes far more total visitors, roughly 2.8 times as many as a 50/50 split. A short ramp-up at low traffic, then 50/50, balances risk and speed.

Answer: A. The sample is too small; wait for a planned sample size

With only 300 sends each, a 3-point difference can easily come from chance. Decide the sample size and duration before the test, then check significance. Peeking and stopping early increases false winners.

Answer: A. Run an experiment that nudges a random group to invite teammates

The pattern may reflect team-based companies that would stay anyway. A randomised nudge tests whether helping users invite teammates causes better retention, before you add friction or redesign around it.

Answer: B. A synthetic control built from other cities

Synthetic control finds weights on untreated units so their combined pre-launch path closely matches the treated unit. After launch, the gap between Dubai and its synthetic version estimates the effect. Placebo runs on other cities show how unusual the gap is.

Answer: D. z ≈ 1.6, not significant

Pooled p = 0.105, so SE = √(0.105 × 0.895 × (1/5,000 + 1/5,000)) ≈ 0.0061. z = 0.01 / 0.0061 ≈ 1.63, and the two-sided p ≈ 0.10, above 0.05. A 10% relative lift can still be real; the test is just underpowered for it.

Answer: A. About 80,000

Required sample size grows with 1/effect², because the standard error falls only with √n. Halving the detectable effect therefore needs about four times as many users. This is why tiny model gains are expensive to prove online.

Answer: B. Regression to the mean: stores chosen for an unusually bad quarter tend to move back toward average anyway, so you need a similar uncoached comparison group

Part of any extreme result is bad luck, and luck does not repeat, so the worst performers usually improve with no treatment at all. Without a control group you cannot separate coaching from that natural bounce. Compare against similar low-performing stores that were not coached, ideally chosen at random.

Answer: A. Self-selection: engaged users are more likely to opt in and also more likely to stay, so the gap overstates the feature effect

Users chose to adopt the feature, and the same motivation that made them adopt it also predicts retention. A randomised test (for example, prompting a random half to turn it on) gives a causal answer. Without one, matching on prior engagement or a before-and-after comparison with a control group reduces the bias.

Answer: A. The changes are confounded: you know the whole package won, but not which change caused it; isolate the copy in a follow-up test if it matters

Bundling changes is fine for shipping a better page, but it does not teach which element mattered. Some parts may even have hurt results. Test individual changes when you need to generalise a lesson.

Answer: D. A novelty effect: returning users explored the new thing out of curiosity, so judge results after behaviour settles or on new users

Change itself attracts attention, which inflates early numbers. The reverse, change aversion, can also make good redesigns look bad at first. Run tests long enough and compare new and returning users separately.

Answer: D. With no real difference, data this extreme would occur about 3% of the time

A p-value is the probability of seeing data at least this extreme if the null hypothesis were true. It is not the probability that the hypothesis is true, nor a measure of effect size. Report effect size and a confidence interval alongside it.

Answer: D. Do not ship yet; investigate why refunds rose, calculate net revenue and customer impact, and adjust the design

Guardrail metrics exist to catch harm a primary metric hides. Higher refunds may mean users were misled or confused into buying. Net impact on revenue, support cost and trust decides whether the change is truly a win.

Answer: C. Keep a long-term holdout group that does not get the shipped changes, and compare it with everyone else over months

Individual test lifts often shrink over time because of novelty effects, overlap between changes, false positives and the winner's curse. A long-running holdout measures the real cumulative impact. It costs some users the improvements, so keep it small.

Answer: B. About 3 points, if both trend alike

The Saudi trend shows what might have happened without the feature. Difference-in-differences subtracts it: 8% - 5% = 3 points. Check that both markets moved in parallel before the launch.

Answer: B. Continue to the planned sample size and at least one or two full weekly cycles, because early results with few conversions are often noise

With only 40 conversions, a 20% difference can easily be random. Repeatedly peeking and stopping when significance first appears inflates false positives. Running to a pre-calculated sample size over full weekly cycles accounts for weekday and weekend behaviour and gives a result you can trust.

Answer: A. With five comparisons, the chance of at least one false winner is about 23%, so adjust for multiple comparisons or confirm with a retest

Each comparison has a 5% false-positive risk, so across five the chance of at least one is 1 - 0.95^5, about 23%. Use corrections like Bonferroni, a stricter threshold, or a follow-up test of the winner. Pre-register the primary metric to avoid cherry-picking.

Answer: A. Make research-led changes and track them; a test would take months

With 9 conversions a month, detecting even a big lift with statistical confidence would take many months. For low-traffic pages, use qualitative research and best practice to make bold improvements, then track the trend.

Answer: A. A random holdout that never gets personalised content

Visitors who get or click personalised content differ from others, for example by being more engaged. Only a randomised holdout, kept on the default experience, shows what personalisation itself adds.

Answer: D. Run ads in matched test cities and hold out similar control cities

A geo-lift test turns ads on in some regions and keeps matched regions as controls, then compares total sales. It needs no user-level tracking. Comparing watchers with skippers is biased because people who watch ads already differ from people who skip.

Answer: D. About 17%

Absolute lift is 2.4% − 2.0% = 0.4 points. As a share of exposed purchases that is 0.4 / 2.4 ≈ 16.7%. So only about one in six purchases in the exposed group was caused by the ads; the rest would have happened anyway.

Answer: A. A novelty effect; people reacted to newness itself

New designs or copy can get attention simply for being different, and the effect fades as people get used to them. Run tests long enough to see past the first reaction and check results by week. Lasting wins come from better messages, not just change.

Answer: A. Compare 90-day retention against a random group shown no offer

Some users would have stayed anyway, and some 'saved' users churn a month later. A randomised holdout shows the extra retention the offer really causes, net of discount cost, over a meaningful period.

Answer: A. A novelty effect, not a lasting change in habit

New things draw attention briefly, so early results can overstate impact. Measuring over a longer period and looking at whether users keep using the feature tells you if the guide built a lasting habit.