41. Upper bound of the fifth retry delay
A client retries with delay = random(0, min(cap, base × 2^attempt)), base = 200 ms, cap = 5,000 ms, attempt counted from 0. What is the largest possible delay at attempt 5?
- A.6,400 ms
- B.5,000 ms
- C.3,200 ms
- D.1,000 ms
Show answer
Answer: B. 5,000 ms
base × 2^5 = 200 × 32 = 6,400 ms, which is above the cap, so the limit is 5,000 ms. The random 'full jitter' spreads clients across 0 to that limit so they do not all retry at the same moment. Without jitter, many clients that failed together retry together.
42. Back-transforming log predictions
You fit a model to log(revenue) and report exp(prediction) as expected revenue per customer. Summed over customers, the forecast comes in well below actual revenue. Why?
- A.The log transform made the model overfit
- B.exp() cannot be applied to negative predictions
- C.exp of the mean of log(y) estimates the median, not the mean, so it underestimates on skewed data
- D.Revenue must be standardised before taking logs
Show answer
Answer: C. exp of the mean of log(y) estimates the median, not the mean, so it underestimates on skewed data
Because exp is convex, exp(E[log y]) is less than E[y] (Jensen). Back-transforming a log model gives something close to the median, which is below the mean for right-skewed revenue. A smearing correction, a variance adjustment or a model that predicts the mean directly, such as Poisson or Tweedie regression, fixes the totals.
43. Average lifetime with active customers
To estimate average customer lifetime, an analyst averages tenure only for customers who have already cancelled, ignoring those still active. What is wrong and what should be used?
- A.Nothing; active customers have no lifetime yet
- B.Active customers are right-censored, so dropping them underestimates lifetime; use survival methods such as Kaplan-Meier
- C.Tenure should be log-transformed before averaging
- D.Active customers should be counted as cancelling today
Show answer
Answer: B. Active customers are right-censored, so dropping them underestimates lifetime; use survival methods such as Kaplan-Meier
Active customers have lived at least as long as their current tenure, which is real information. Dropping them keeps mostly short-lived customers, and treating them as cancelling today also cuts lifetimes short. Survival analysis (Kaplan-Meier curves, Cox models) uses censored observations correctly.
44. Making a simulation ten times more precise
A Monte Carlo simulation with 10,000 runs estimates a loss probability with a standard error of 0.004. How many runs are needed for a standard error of 0.0004?
- A.100,000
- B.20,000
- C.10,00,000 (one million)
- D.1,00,00,000 (ten million)
Show answer
Answer: C. 10,00,000 (one million)
Simulation error shrinks with the square root of the number of runs. Cutting it by a factor of 10 needs 10² = 100 times more runs: 10,000 × 100 = 1,000,000. Variance-reduction tricks can help when that is too costly.
45. Two trending series that correlate
Regressing yearly smartphone users on yearly tea exports gives R² = 0.93 and a tiny p-value. Both have risen steadily for 15 years. What is the likely problem?
- A.The sample is too large
- B.Spurious regression from shared trends
- C.Tea exports cause smartphone use to grow
- D.R² is always high for yearly data
Show answer
Answer: B. Spurious regression from shared trends
Two unrelated series with trends will look strongly correlated, and standard p-values assume stationary errors, so they mislead. Regress differences or growth rates instead, or test for cointegration. The high R² reflects time, not a link.
46. Choosing the AR order
A stationary series has a partial autocorrelation plot with significant spikes at lags 1 and 2 and nothing after. The ACF decays slowly. Which model does this suggest?
- A.MA(2)
- B.AR(1)
- C.AR(2)
- D.White noise
Show answer
Answer: C. AR(2)
For an autoregressive AR(p) process, the PACF cuts off after lag p while the ACF tails off. A cutoff after lag 2 points to AR(2). For a moving-average MA(q) process it is the reverse: the ACF cuts off after lag q.
47. Ranking many prompts from pairs
You have pairwise judge results for 8 prompt variants, but not every pair was compared equally often. What is a principled way to turn these into one ranking?
- A.Fit a Bradley–Terry (Elo-style) strength model
- B.Rank them by the raw number of wins only
- C.Rank by the length of each prompt
- D.Pick the variant that was compared most often
Show answer
Answer: A. Fit a Bradley–Terry (Elo-style) strength model
A Bradley–Terry model estimates a strength score per variant from who beat whom, accounting for opponents faced and uneven pair counts. Raw win counts favour variants that happened to face weak opponents or played more often.
48. No failures seen in 150 tests
A red-team suite of 150 independent prompts produced zero harmful outputs. Using the "rule of three", what is the approximate 95% upper bound on the true harmful-output rate?
- A.About 2%
- B.0%
- C.About 0.2%
- D.About 6.7%
Show answer
Answer: A. About 2%
With zero events in n trials, the 95% upper bound is about 3 ÷ n = 3 ÷ 150 = 2%. Seeing no failures does not mean the rate is zero; it only rules out rates much above 2% for this kind of input.
49. Overall rate goes the other way
Version B has a higher conversion rate than A on both mobile and desktop, but A has a higher conversion rate overall. What explains this?
- A.The test must have a bug, because this is mathematically impossible
- B.B received a much larger share of low-converting mobile traffic, so mixing segments reverses the overall result (Simpson's paradox)
- C.The p-value is too high
- D.Desktop users are counted twice
Show answer
Answer: B. B received a much larger share of low-converting mobile traffic, so mixing segments reverses the overall result (Simpson's paradox)
This is Simpson's paradox: when groups have very different sizes and base rates, combining them can reverse a trend seen in every subgroup. Here B got more traffic from the weaker segment. Check that randomisation gave a balanced mix and compare like with like.
50. How reliable is a fraud flag
1% of transactions are fraud. A model flags 99% of fraud cases and also wrongly flags 5% of normal transactions. If a transaction is flagged, about how likely is it to be fraud?
- A.About 17%
- B.About 95%
- C.About 99%
- D.About 50%
Show answer
Answer: A. About 17%
Out of 10,000 transactions, 100 are fraud and about 99 are flagged. Of the 9,900 normal ones, 5% = 495 are also flagged. So only 99 / (99 + 495), about 17%, of flags are real fraud. When the base rate is low, even a good model produces mostly false alarms.
51. Halving the margin of error
A survey of 400 people gives a margin of error of plus or minus 5 points. Roughly how many people do you need to cut it to plus or minus 2.5 points?
- A.800
- B.1,200
- C.1,600
- D.4,000
Show answer
Answer: C. 1,600
Margin of error shrinks with the square root of the sample size. To halve it you need square root of n to double, which means n must grow by four times: 400 x 4 = 1,600. This is why each extra bit of precision gets expensive.
52. Worst stores improved after coaching
A dashboard shows that the 10 worst-scoring stores last quarter, which got coaching, improved this quarter, while the 10 best got worse. What else could explain this?
- A.The data must be wrong for both groups
- B.Coaching harmed the best stores
- C.Regression to the mean, as extremes include luck
- D.The scores were rounded differently each quarter
Show answer
Answer: C. Regression to the mean, as extremes include luck
Extreme results partly reflect chance, so the next measurement tends to be closer to average even with no action. To credit coaching, compare with a similar group of low-scoring stores that did not get it.
53. SUS 72 versus 68 with 10 users each
The new design scores a mean SUS of 72 and the old one 68, with 10 different participants each. SUS standard deviations are typically around 17. What is the best conclusion?
- A.The new design is clearly better by 4 points
- B.The old design is clearly better
- C.The gap is well within noise for this sample size
- D.SUS cannot be compared between groups
Show answer
Answer: C. The gap is well within noise for this sample size
With SD ≈ 17 and 10 per group, the standard error of the difference is about √(2 × 17² ÷ 10) ≈ 7.6 points, so a 4-point gap gives t ≈ 0.5, far from significant. You would need far more participants to detect a difference this small.
54. How many users an A/B test needs
Your checkout converts at 5%. You want to detect a lift to 6% with 80% power at 5% significance. Using the rule of thumb n ≈ 16 × p(1 − p) ÷ δ² per variant, roughly how many users does each variant need?
- A.About 760
- B.About 7,600
- C.About 76,000
- D.About 150
Show answer
Answer: B. About 7,600
p(1 − p) = 0.05 × 0.95 = 0.0475 and δ = 0.01, so n ≈ 16 × 0.0475 ÷ 0.0001 = 7,600 per variant. Small absolute lifts on low base rates need large samples, which is why low-traffic products often rely on qualitative tests instead.
55. What CUPED does
Your experimentation platform offers CUPED. What does it do?
- A.Corrects tests for a sample ratio mismatch automatically
- B.Lets you change the primary metric after the test
- C.Uses pre-test behaviour to cut noise, so tests need fewer users
- D.Removes the need for random assignment of users
Show answer
Answer: C. Uses pre-test behaviour to cut noise, so tests need fewer users
CUPED (Controlled-experiment Using Pre-Experiment Data) adjusts each user's metric using a correlated pre-test value, such as past spend. This lowers variance, so tests reach the same power with smaller samples or in less time.
56. Diluted effect from untouched users
A test changes only the checkout page, but all site visitors are counted. Only 10% of visitors reach checkout. If checkout conversion rises 10%, what happens to the site-wide measured effect?
- A.It shows the full 10% lift for every visitor
- B.It increases, because more visitors are counted
- C.It is diluted by the 90% who never saw the change
- D.It reverses, showing a loss across the site
Show answer
Answer: C. It is diluted by the 90% who never saw the change
Users who never reach the changed page add noise but no effect, so the overall lift looks much smaller and needs a far larger sample to detect. Analyse users who were exposed (triggered) to the change, assigned at that point.
57. What p = 0.03 does not mean
A test returns p = 0.03. Which statement is correct?
- A.With no true effect, data this extreme would occur about 3% of the time
- B.There is a 97% chance that the variant is truly the better one
- C.There is a 3% chance that the variant is truly the better one
- D.The variant improved the conversion rate by 3 percentage points
Show answer
Answer: A. With no true effect, data this extreme would occur about 3% of the time
A p-value is the probability of seeing data at least this extreme if the null hypothesis is true. It is not the probability that the hypothesis is true or false, and it says nothing about effect size.
58. Is a 3% vs 2.5% click gap real?
Version A was sent to 2,000 people and got 50 clickers (2.5%). Version B went to 2,000 and got 60 (3.0%). Using a standard two-proportion z-test, what can you conclude?
- A.Not significant; z is about 1, well below 1.96
- B.Significant at 95%; B is the clear winner
- C.Significant at 99%, as the gap is 20% relative
- D.Cannot be tested, as the sample sizes are equal
Show answer
Answer: A. Not significant; z is about 1, well below 1.96
Pooled rate is 110 ÷ 4,000 = 2.75%. Standard error = √(0.0275 × 0.9725 × (1/2,000 + 1/2,000)) ≈ 0.0052, and z = 0.005 ÷ 0.0052 ≈ 0.97. That is far from 1.96, so the difference could easily be noise.
59. Comparing conversion on small samples
Article A converts 10 of 500 visitors (2%). Article B converts 3 of 100 (3%). A colleague wants to rewrite A to copy B. What is the best response?
- A.B's sample is too small; 3 conversions could easily be chance
- B.B is clearly 50% better, so copy it right away
- C.A is better, because it has more total conversions
- D.Both rates are equal once you round them to whole numbers
Show answer
Answer: A. B's sample is too small; 3 conversions could easily be chance
With only 3 conversions, B's rate is very uncertain: one more or one fewer changes it by a full point. Before acting, collect more data or look for a consistent pattern across several pages. Small samples often produce big-looking differences that disappear.
60. 'The average salary is'
A draft says 'The average salary at the startup is ₹40 lakh, so most staff earn well.' The founders earn ₹3 crore each and most staff earn ₹8-12 lakh. What should the editor flag?
- A.Nothing; an average always means the typical salary
- B.The figure should be shown in US dollars for clarity
- C.Change 'most staff earn well' to 'all staff earn well'
- D.A few huge salaries pull up the mean; the median fits better
Show answer
Answer: D. A few huge salaries pull up the mean; the median fits better
A mean can be distorted by a few extreme values, so it does not describe a typical employee here. The median, the middle value, would be close to ₹10 lakh and would contradict 'most staff earn well'. Editors should check which average is meant and whether the conclusion follows.
