After deploying a model, what would you monitor, and how do you handle data drift?
What the interviewer wants to hear
- System metrics: latency, errors, throughput
- Input data drift: feature distributions, missing values
- Prediction drift and, when labels arrive, real performance
- Alerts with thresholds and dashboards
- Retraining triggers and safe rollout
Model answer
I monitor at three levels. First, system health: latency, error rate, throughput and resource use, like any service. Second, data: distributions of each input feature, missing values, and new categories, compared against the training data using measures like population stability index or KL divergence. I also track the distribution of predictions, since a sudden shift often means an upstream bug. Third, model quality: once labels arrive, I compute the real business metric and model metrics over time, sliced by important segments. When drift appears, I first check whether it is a pipeline bug, such as a changed unit or a broken join. If the world has truly changed, I retrain on recent data, validate against the current model, and roll out gradually with a canary or shadow deployment. Scheduled retraining plus alerts covers most cases.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Machine Learning Engineer questions
- You are building a fraud model where 0.5% of transactions are fraud. Walk through how you would train and evaluate it.
- What problems does a feature store solve, and when is it worth adding one?
- How do you decide between batch predictions and real-time (online) inference for a model?
- Your neural network loss is not decreasing. How do you debug it?
- Walk me through how you would build a RAG system to answer questions over a company knowledge base.
- How would you evaluate an LLM-powered feature before and after launch?
- How do you approach hyperparameter tuning for a model on a limited compute budget?
- A team wants an LLM to answer questions about their internal products. How do you decide between prompting, RAG and fine-tuning?
- Your new model scores far higher than anything before it. How do you check for data leakage?