You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?

What the interviewer wants to hear

• Frequent, complete, asynchronous checkpoints to durable storage
• Automatic restart and elastic or fixed-size resume logic
• Handle preemption notices and save on shutdown
• Watch for stragglers and failing nodes
• Track cost and goodput, not just GPU hours

Model answer

The core is checkpointing. I save complete state, meaning model and optimiser shards, scheduler, step, random states and data loader position, to object storage on a fixed interval, chosen so the expected lost work per preemption is small compared with checkpoint cost. Asynchronous or sharded checkpointing keeps saves from stalling training. I listen for preemption notices and trigger a final save when one arrives. The orchestrator restarts the job automatically on new nodes and resumes from the latest valid checkpoint, verifying it is complete before using it. If node counts can change, I use elastic training or keep the global batch size constant through gradient accumulation. I monitor for stragglers and bad nodes that slow every step. Finally I measure goodput, the share of GPU time that produced progress, so I can compare the spot savings against time lost to restarts.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• How would you self-host and serve an open-weight LLM for thousands of concurrent users?
• Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.
• The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?
• Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?
• How do you build a real-time feature pipeline whose values match the batch features used in training?