Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?

What the interviewer wants to hear

• Measure time to first token and per-token latency separately
• Reduce prompt size and use prefix caching for shared prompts
• Tune batching and concurrency limits for the latency target
• Use quantisation, speculative decoding or a smaller model where quality allows
• Stream responses and evaluate quality after every change

Model answer

First I measure where the time goes. Time to first token is mostly prefill and queueing, while total time is driven by decode speed and output length. If prompts are long, I trim them, cut retrieved context and enable prefix caching so shared system prompts are not recomputed. If queueing dominates, I add capacity or lower the maximum batch size, because large batches raise throughput but slow each request. For decode speed, I try quantised weights, speculative decoding with a small draft model, and tensor parallelism across more GPUs, which lowers per-token latency at some cost. I also look at product choices: shorter outputs, streaming tokens to the user, and routing simple requests to a smaller model. Every change goes through the same quality evaluation set and a load test, so I can see the latency, throughput and quality trade-off before shipping.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• How would you self-host and serve an open-weight LLM for thousands of concurrent users?
• Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.
• The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?
• Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?
• How do you build a real-time feature pipeline whose values match the batch features used in training?