How would you estimate the GPUs needed to serve an open-weight LLM for a given number of users?

What the interviewer wants to hear

• Weights memory from parameters times bytes per parameter
• KV cache memory per token times context length times concurrent sequences
• Leave headroom for activations and runtime overhead
• Measure tokens per second and latency under realistic load
• Consider quantisation, GQA and batching to change the numbers

Model answer

I start with weight memory: parameters times bytes per parameter, so a 70B model in 16-bit needs about 140 GB, and 4-bit cuts that to roughly 35 GB. Then I estimate the KV cache: 2 times layers times KV heads times head size times bytes, per token, multiplied by the typical context length and the number of sequences in flight. With grouped-query attention this can be several times smaller than the naive estimate. I add headroom for activations and runtime overhead, usually 10 to 20 percent. That tells me how many GPUs are needed just to fit the model and the concurrency, and whether tensor parallelism is needed. Then I benchmark with a realistic mix of prompt and output lengths to measure tokens per second and time to first token, and size the fleet for peak traffic with margin.

More medium MLOps Engineer questions

• What does a good CI/CD pipeline for machine learning look like?
• Explain shadow, canary and A/B deployments for models. When do you use each?
• Design an automated retraining pipeline for a model whose data changes weekly.
• What operational challenges come with running a feature store, and how do you handle them?
• How do you run GPU model inference efficiently on Kubernetes?
• How do you design a training pipeline in an orchestrator like Airflow so that it is reliable and easy to rerun?
• How would you define service level objectives for a production model service?
• How do you secure the ML pipeline and model supply chain?
• What would you monitor for an LLM-powered application in production, beyond normal service metrics?