How would you self-host and serve an open-weight LLM for thousands of concurrent users?

What the interviewer wants to hear

• Serving engine with continuous batching and paged KV cache
• GPU memory planning: weights, KV cache, quantisation
• Parallelism across GPUs if model is large
• Autoscaling, routing, rate limiting, streaming
• Track TTFT, tokens per second, cost; evaluate quality after optimisations

Model answer

I would use a dedicated serving engine such as vLLM or a similar runtime, because continuous batching and paged KV cache management give far higher throughput than a simple web server. First I plan GPU memory: the weights plus enough KV cache for expected context lengths and concurrency. Quantisation to 8-bit or 4-bit can cut memory and cost, but I check quality on an eval set. If the model does not fit on one GPU, I use tensor parallelism within a node. I put a gateway in front for authentication, rate limits per tenant, request routing and streaming responses. Autoscaling uses queue depth and GPU metrics, with warm replicas because loading weights is slow. Prefix caching helps when many requests share a system prompt. I monitor time to first token, tokens per second, error rate and cost per million tokens.

More hard MLOps Engineer questions

• Design an internal ML platform for 15 data science teams who currently deploy models in different ad hoc ways.
• Your recommendation model suddenly shows a 25% drop in click-through rate. Walk through your incident response.
• The ML infrastructure bill doubled in six months. How do you bring it down without slowing teams?
• A regulator asks you to show exactly how a credit decision model was built and how a past decision was made. How should your MLOps setup support this?
• You need to run a multi-day, multi-node training job on preemptible GPUs. How do you make it reliable and cost-effective?
• Your RAG system has 200 million chunks indexed with an old embedding model. How do you move to a new embedding model without downtime or quality loss?
• Users say a self-hosted LLM feature is too slow. How do you reduce latency without destroying throughput or quality?
• Under privacy law, a customer asks you to delete all of their data. How should an ML platform handle this across datasets, features, indexes and models?
• How do you build a real-time feature pipeline whose values match the batch features used in training?