A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?

What the interviewer wants to hear

• Profile end to end: model, features, network, preprocessing
• Quantisation, pruning, distillation to a smaller model
• Compile and optimise with runtimes such as ONNX Runtime or TensorRT
• Batching, caching, and right hardware
• Measure accuracy loss for each change

Model answer

First I profile the full request path, because often the slow part is feature lookups or preprocessing, not the model. For the model itself, I try quantisation to int8 or lower precision, which can give large speedups with small accuracy loss. Knowledge distillation trains a smaller student model to copy the large one, often keeping most of the quality. Pruning can help on supported hardware. I export to an optimised runtime like ONNX Runtime or TensorRT, which fuses operations. On the serving side, dynamic batching improves GPU use, caching handles repeated inputs, and choosing the right instance type or moving small models to CPU can cut cost. Sometimes a cheaper model handles most requests and only hard cases go to the large one. For every change I measure accuracy on the validation set and latency at p95 and p99, not just average.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?
• You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?
• How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?
• How would you design LLM inference serving to maximise throughput while keeping latency acceptable?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?
• Design a real-time fraud detection system for card payments.
• You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?