What would you monitor for a model running in production?

What the interviewer wants to hear

• Service health: latency, errors, throughput, resource use
• Data quality: missing values, schema changes
• Data and prediction drift
• Model performance when labels arrive; business KPIs

Model answer

I monitor four things. First, service health, like any API: request rate, latency at p50, p95 and p99, error rate, and CPU, memory or GPU use. Second, data quality: schema changes, missing values, values outside expected ranges and new categories. These often catch upstream bugs early. Third, drift: compare feature distributions and the prediction distribution with the training data or a recent baseline. A sudden jump in the share of positive predictions is a strong warning. Fourth, model and business performance: when true labels arrive, compute metrics like precision or error over time and by segment, and track the business KPI the model supports. All of this goes to dashboards with alerts that have sensible thresholds and owners, so the right person acts when something goes wrong.

More easy MLOps Engineer questions

• What is MLOps, and how is it different from traditional DevOps?
• Why does a team need a model registry? What information should it store?
• How do you make a model training run reproducible?
• How would you deploy a trained scikit-learn model as a REST API?
• What is training-serving skew, and how do you prevent it?
• How do you decide between batch predictions and a real-time model API?
• Explain the difference between data drift and concept drift, and how you would detect each.
• What makes a good Docker image for a model-serving service?
• Which Kubernetes concepts matter most when you deploy a model service, and why?