You must self-host an open-weights LLM for a high-traffic product. How do you plan memory, throughput and latency?

What the interviewer wants to hear

• Size weights by parameter count and precision, and choose quantization carefully
• Budget KV cache memory from layers, KV heads, context and concurrency
• Use an inference server with continuous batching and paged KV cache
• Balance throughput against time to first token and per-token latency
• Load test with realistic prompt and output lengths, then autoscale

Model answer

I start with memory. Weights need parameters times bytes per parameter, so a model in 16-bit needs about 2 bytes per parameter, and 8-bit or 4-bit quantization cuts that but must be checked on our evals. Then the KV cache: per token it is 2 × layers × KV heads × head size × bytes, multiplied by context length and concurrent sequences, and it often dominates at long contexts. I would use a serving engine with continuous batching and paged KV cache, which raises throughput a lot. The key trade-off is throughput versus latency: bigger batches lower cost per token but raise time per token, so I set limits from our latency targets. Prefix caching and speculative decoding can help further. Finally I load test with realistic prompt and output lengths, and autoscale on queue depth.

More hard AI Engineer (LLM & Generative AI) questions

• Design a RAG assistant over 2 million internal documents with different access permissions per team.
• How would you protect an LLM application that reads emails and can take actions, against prompt injection and data leaks?
• An agent that completes multi-step tasks works in demos but fails often in production. How do you make it reliable?
• Your LLM feature costs too much and p95 latency is 9 seconds. Walk through how you would reduce both without hurting quality.
• When would you fine-tune a smaller open-weight model instead of using a large hosted model, and how would you do it?
• Users say your AI assistant got worse this week, but nothing obvious changed. How do you investigate?
• How do you design an LLM-as-judge evaluation you can actually trust?
• When would you choose a multi-agent architecture over a single agent or a fixed workflow?
• Your agent will connect to several MCP servers, some built by third parties. How do you make this secure?