What are the training, validation and test sets used for, and how do you split them?

What the interviewer wants to hear

• Training set fits parameters
• Validation set tunes hyperparameters and picks models
• Test set gives a final unbiased estimate, used once
• Split respecting time, groups and class balance (stratify)

Model answer

The training set is what the model learns its parameters from. The validation set is used to compare models and tune hyperparameters such as learning rate or tree depth. The test set is held back until the very end and used once to estimate how the final model will perform on new data. If I keep checking the test set while tuning, it stops being unbiased. How I split depends on the data. For classification with imbalanced classes I use a stratified split so each set has similar class ratios. For time-based data I split by time, training on the past and testing on the future. If the same user or patient appears many times, I split by group so the same person is not in both train and test, which would leak information.

More easy Machine Learning Engineer questions

• What is overfitting, how do you detect it, and how do you reduce it?
• Explain precision and recall with a real example. When would you favour one over the other?
• How does gradient descent work, and what is the role of the learning rate?
• Why should you build a simple baseline before a complex model? Give examples of good baselines.
• Explain the bias-variance trade-off and how it guides your choice of model.
• What is an embedding, and how would you use embeddings in a product feature?
• When would you choose a tree-based model like gradient boosting over a linear model, and when not?
• How do you handle missing values in a dataset before training a model?
• Explain in simple terms how self-attention works in a transformer.