The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?

What the interviewer wants to hear

• Visibility: cost attribution per team, model and job
• Training: spot or preemptible instances with checkpointing, right-sizing
• Serving: utilisation, batching, autoscaling, scale to zero, model compression
• Clean up idle resources, storage and unused features
• Guardrails: quotas and budgets, not blocking approvals

Model answer

I start with visibility: tag every resource by team, project and model, and build a cost dashboard. Usually a few items explain most of the spend. For training, I move fault-tolerant jobs to spot or preemptible instances with regular checkpointing, right-size instance types, and stop idle notebooks and development GPUs automatically. For serving, I look at utilisation: many GPU services run at 10 to 20 percent. Batching, better autoscaling, sharing GPUs between small models, moving small models to CPU, and scale to zero for rarely used endpoints all help. Quantisation or distillation can cut the hardware needed. I also clean up old artifacts, logs and unused features. To keep teams fast, I use budgets, quotas and alerts with clear ownership instead of manual approvals, and share the savings each team achieved.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• How would you self-host and serve an open-weight LLM for thousands of concurrent users?
• Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?
• Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?
• Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?
• How do you build a real-time feature pipeline whose values match the batch features used in training?