How would you design LLM inference serving to maximise throughput while keeping latency acceptable?

What the interviewer wants to hear

• Continuous batching to keep the GPU busy
• Paged KV cache and prefix caching to fit more requests
• Quantisation and efficient attention kernels
• Speculative decoding or smaller models for latency
• Separate SLOs for time to first token and per-token latency
• Autoscaling, routing and load testing with realistic traffic

Model answer

I would start with a modern serving engine that supports continuous batching, so new requests join the running batch at each decoding step instead of waiting for a whole batch to finish. Memory is the main limit, so I use a paged KV cache to avoid fragmentation and prefix caching for shared system prompts. Quantising weights to 8 or 4 bits, and the KV cache where quality allows, frees memory for more concurrent sequences. Fused attention kernels cut memory traffic. For latency I track time to first token and time per output token separately, cap the maximum batch size to protect tail latency, and consider speculative decoding. I might route easy requests to a smaller model. Finally I load test with realistic prompt and output lengths, set autoscaling on queue depth and KV cache usage, and monitor p50, p99 and cost per thousand tokens.

More hard Machine Learning Engineer questions

• Design a recommendation system for an e-commerce home page with 10 million products and 50 million users.
• A model performs much worse in production than in offline tests. How do you find and prevent training-serving skew?
• You need to train a model that is too large for one GPU and takes weeks on one machine. How do you scale training?
• How would you run an A/B test to decide whether a new ML model should replace the current one? What pitfalls do you watch for?
• A deep learning model meets accuracy targets but is too slow and expensive to serve. What options do you consider?
• What guardrails would you put around an LLM agent that can call internal tools and read customer data?
• Scaling training from 8 to 64 GPUs gave only a 3x speed-up. How do you find and fix the bottleneck?
• Design a real-time fraud detection system for card payments.
• You need to cut the serving cost of a large model by quantising it. How do you do it without hurting quality?