Why do we split data into training, validation and test sets? How do you decide how to split?

What the interviewer wants to hear

• Training data fits the model parameters
• Validation data is used for tuning and model choice
• Test data gives a final unbiased estimate and is used once
• Split by time for forecasting and by entity to avoid leakage
• Cross-validation helps when data is small

Model answer

The training set is what the model learns from. The validation set is used to compare models and tune hyperparameters, and because we make many choices with it, its score becomes slightly optimistic. The test set is kept aside and used once at the end to estimate how the model will perform on truly new data. How I split depends on how the model will be used. If it predicts the future, I split by time, training on older data and testing on newer data. If one customer has many rows, I split by customer so the same person never appears in both training and test, otherwise the model can memorise them. For small datasets I use k-fold cross-validation for tuning to get more stable estimates, but still keep a final holdout when possible.

More easy Data Scientist questions

• Explain what a p-value is to a product manager, and one common way it is misunderstood.
• When would you report the median instead of the mean? Give a business example.
• What is the difference between correlation and causation? How would you test if a change causes an outcome?
• How do you handle missing values in a dataset?
• How do you present a data analysis to non-technical stakeholders?
• Explain the trade-off between precision and recall, and how you would pick a threshold for a real model.
• What is the central limit theorem and why does it matter for everyday analysis work?
• You receive a new, messy dataset for an analysis. What are your first steps before any modelling?
• How do you choose the right chart to present a finding to senior leaders?