You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?

What the interviewer wants to hear

• Data parallelism for speed when the model fits
• Sharding (FSDP/ZeRO), tensor and pipeline parallelism when it does not
• Mixed precision and activation checkpointing for memory
• Communication overhead, interconnect and batch size effects
• Checkpointing and fault tolerance

Model answer

I first profile to find the real bottleneck: compute, memory or input pipeline. For memory, mixed precision with bf16 and activation checkpointing are cheap wins. If the model still does not fit, I use fully sharded data parallel or ZeRO, which shards parameters, gradients and optimizer states across GPUs. For very large models I add tensor parallelism within a node, where interconnect is fast, and pipeline parallelism across nodes. For speed, data parallelism across many GPUs scales throughput, but a larger global batch may need learning rate warmup and tuning to keep quality. Communication becomes the limit at scale, so I overlap it with compute and use fast interconnects. Long runs fail, so I save sharded checkpoints regularly, make jobs resumable, and track loss and gradient norms to catch instability early.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?
• How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?
• A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?
• How would you design LLM inference serving to maximise throughput while keeping latency acceptable?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?
• Design a real-time fraud detection system for card payments.
• You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?