How would you validate a demand forecasting model before the business relies on it?

What the interviewer wants to hear

• Use rolling-origin backtests that respect time order
• Compare against simple baselines such as seasonal naive
• Pick error metrics that match the business, such as MAPE, WAPE or bias
• Check features only use data available at forecast time
• Review errors by segment, horizon and special periods

Model answer

I would run a rolling-origin backtest: train on data up to a cutoff, forecast the next horizon, move the cutoff forward and repeat, so every test period is in the future relative to training. I would always compare against simple baselines like last year same week or a seasonal naive forecast, because a complex model that cannot beat them is not worth running. For metrics I would use WAPE or MAE for accuracy and check bias separately, since a forecast that is always 5% low causes stock-outs even if its error looks small. I would check every feature is available at forecast time, for example using promotions that are planned in advance but not actual sales. Finally I would break errors down by product, store, horizon and holiday periods, and share those results with planners so they know where to trust the forecast.

More medium Data Scientist questions

• Walk through how you would design an A/B test for a new checkout page.
• How would you choose a success metric for a new "recommended for you" section in a shopping app?
• You have 500 candidate features for a churn model. How do you decide which ones to keep?
• A business team does not trust your credit risk model. How do you explain its predictions?
• The CEO asks: "Why are customers churning?" How do you approach this?
• You are building a model where only 0.5% of examples are positive. How do you approach training and evaluation?
• What is data leakage, how do you detect it, and how do you prevent it in a modelling project?
• How do you decide between a regularised linear model and a gradient boosted tree model for a tabular problem?
• Walk me through how you would decide the sample size and duration of an A/B test.