A feature was launched to all users at once, so there is no control group. How would you estimate its impact?

What the interviewer wants to hear

• Acknowledge a simple before-after comparison is biased by trends and seasonality
• Difference-in-differences using an unaffected group or region
• Interrupted time series or forecasting the counterfactual
• Compare adopters vs non-adopters carefully with matching, noting selection bias
• State assumptions and uncertainty clearly

Model answer

A simple before versus after is risky because seasonality, marketing campaigns or general growth can explain the change. So I try to build a believable counterfactual, meaning what would have happened without the feature. If some group was not affected, like a region, platform or an older app version, I use difference-in-differences: compare the change in the affected group with the change in the unaffected group, after checking their trends were parallel before launch. If everyone got it, I can use an interrupted time series: model the historical trend and seasonality, forecast the post-launch period, and compare actuals against the forecast. Comparing users who used the feature with those who did not is tempting but has selection bias, since engaged users adopt features first; matching on past behaviour helps somewhat. I present a range, not a single number, and clearly state the assumptions.

More hard Data Analyst questions

• An A/B test shows a significant lift in conversion but a drop in average order value, and revenue per user is flat. What do you recommend?
• Finance and Marketing report different revenue numbers for the same month. How do you resolve it?
• Your team gets 50 ad-hoc data requests a week. How would you reduce that load without hurting the business?
• An experiment report checks 20 metrics across 10 segments and highlights the 3 that are significant at p < 0.05. How do you judge it?
• The CFO asks you to forecast next quarter revenue. How do you approach it, and how do you communicate uncertainty?
• A senior leader asks you to "find data that proves" their new campaign worked. Your analysis suggests it did not. What do you do?
• You run an A/B test on a ride-hailing or delivery marketplace where riders and drivers share the same supply. Why might a normal user-level test give a wrong answer, and what would you do?
• Marketing wants to segment customers for targeted campaigns. How would you build the segmentation, and how would you know it is useful?
• Your executive dashboard was wrong twice last quarter and leaders have stopped trusting it. How do you rebuild trust?