How do you run GPU model inference efficiently on Kubernetes?

What the interviewer wants to hear

• Device plugin and GPU resource requests
• Node pools, taints and tolerations for GPU nodes
• Batching and right model server for high utilisation
• Autoscaling on suitable metrics; cold start handling
• GPU sharing options and cost monitoring

Model answer

I set up dedicated GPU node pools with the NVIDIA device plugin, and taint those nodes so only pods that tolerate the taint and request nvidia.com/gpu land there. That stops ordinary workloads from wasting expensive nodes. For the service itself, I use a proper inference server, like Triton or vLLM for LLMs, with dynamic or continuous batching to keep the GPU busy. Autoscaling on CPU usage does not work well for GPU services, so I scale on queue length, concurrency or GPU utilisation through custom metrics. Because GPU pods start slowly, I reduce image size, cache model weights near the node, and keep a minimum number of warm replicas. For small models, GPU sharing such as MIG or time slicing improves use. I monitor utilisation and cost per thousand predictions to keep spend under control.

More medium MLOps Engineer questions

• What does a good CI/CD pipeline for machine learning look like?
• Explain shadow, canary and A/B deployments for models. When do you use each?
• Design an automated retraining pipeline for a model whose data changes weekly.
• What operational challenges come with running a feature store, and how do you handle them?
• How do you design a training pipeline in an orchestrator like Airflow so that it is reliable and easy to rerun?
• How would you estimate the GPUs needed to serve an open-weight LLM for a given number of users?
• How would you define service level objectives for a production model service?
• How do you secure the ML pipeline and model supply chain?
• What would you monitor for an LLM-powered application in production, beyond normal service metrics?