Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?

What the interviewer wants to hear

• Profile compute, communication and data loading separately
• Check the data pipeline keeps GPUs fed
• Look at all-reduce or all-gather time and network bandwidth
• Overlap communication with compute and increase work per step
• Watch for stragglers, load imbalance and small per-GPU batches

Model answer

First I measure instead of guessing. I profile a few steps on 8 and 64 GPUs and break step time into data loading, forward and backward compute, and communication. If GPU utilisation is low and there are gaps before each step, the data loader is the bottleneck, and I add workers, prefetching or faster storage. If communication dominates, I look at gradient all-reduce or FSDP all-gather time. Across nodes the network is much slower than within a node, so I check interconnect bandwidth, make sure communication overlaps with backward compute, use bucketed gradients, and increase work per step through larger per-GPU batches or gradient accumulation. I also look for stragglers: one slow node or uneven sequence lengths make everyone wait. After each change I re-measure scaling efficiency and keep the learning rate tuned for the new global batch.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?
• You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?
• How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?
• A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?
• How would you design LLM inference serving to maximise throughput while keeping latency acceptable?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Design a real-time fraud detection system for card payments.
• You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?