Your new model scores far higher than anything before it. How do you check for data leakage?
What the interviewer wants to hear
- Be suspicious of results that seem too good
- Inspect top features for ones that encode the label or future data
- Check that splits respect time and groups such as users
- Ensure preprocessing is fitted only on training data
- Confirm with a clean, later holdout or a shadow test
Model answer
A sudden big jump makes me suspicious before it makes me happy. I first look at feature importance or SHAP values and examine the top features: is any of them created after the outcome, such as a "refund issued" flag in a churn model, or a proxy for the label? Next I check the split. For time-based problems, training data must come strictly before validation data, and rows from the same user or entity must not appear on both sides. I look for duplicate or near-duplicate rows across splits. Then I check the pipeline: scalers, imputers, target encoders and resampling must be fitted only on training folds. Finally I score the model on a clean holdout from a later period, or run it in shadow mode on live traffic. If the gain disappears there, it was leakage.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Machine Learning Engineer questions
- You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
- What problems does a feature store solve, and when is it worth adding one?
- How do you decide between batch predictions and real-time (online) inference for a model?
- Your neural network loss is not decreasing. How do you debug it?
- After deploying a model, what would you monitor, and how do you handle data drift?
- Walk me through how you would build a RAG system to answer questions over a company knowledge base.
- How would you evaluate an LLM-powered feature before and after launch?
- How do you approach hyperparameter tuning for a model on a limited compute budget?
- A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?