Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.

What the interviewer wants to hear

• Triage impact and communicate; consider rollback or fallback quickly
• Check recent changes: deployments, data pipelines, features, configs
• Inspect data quality and drift on inputs and predictions
• Find root cause and fix
• Post-incident review with preventive actions

Model answer

First I confirm the drop is real and not a tracking problem, and estimate the impact. I open an incident, tell stakeholders, and if impact is high I mitigate fast: roll back to the previous model version, or switch to a safe fallback like popularity-based recommendations, while investigating. Next I check what changed: model deployments, feature pipeline releases, upstream data sources, config flags and experiments. Most incidents are data problems, so I compare live feature distributions and the prediction distribution with last week. For example, a feature might suddenly be all null because an upstream table was renamed, and the model silently uses default values. Once the root cause is found, I fix it, backfill if needed, and redeploy carefully. Afterwards I write a blameless postmortem and add prevention, such as schema checks and alerts on feature null rates.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• How would you self-host and serve an open-weight LLM for thousands of concurrent users?
• The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?
• Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?
• Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?
• How do you build a real-time feature pipeline whose values match the batch features used in training?