How do you plan and carry out a Kubernetes version upgrade for production clusters with minimal risk?
What the interviewer wants to hear
- Read release notes; find deprecated and removed APIs
- Upgrade one minor version at a time; control plane before nodes
- Test in lower environments first; check add-on compatibility
- PodDisruptionBudgets and surge node pools for safe draining
- Rollback plan, such as blue-green clusters, and monitoring
Model answer
I start with the release notes and scan all manifests and Helm charts for deprecated or removed APIs using tools like Pluto or kubent, then fix them before upgrading. I check that add-ons such as the CNI, ingress controller, CSI drivers, service mesh and operators support the new version. Kubernetes supports upgrading one minor version at a time and has version skew rules, so the control plane goes first, then node pools. I upgrade dev and staging first and run tests. For nodes, I create a new node pool on the new version, cordon and drain old nodes gradually, and rely on PodDisruptionBudgets, multiple replicas and good readiness probes so services stay available. I watch error rates and cluster health during the process. For very critical systems, I prefer a blue-green cluster approach, moving traffic to a new cluster so rollback is simple.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard DevOps Engineer questions
- You are asked to build an internal platform so 30 product teams can deploy services to Kubernetes on their own. How would you design it?
- How would you secure the software supply chain from source code to a running container in production?
- Your company wants to run a critical service active-active in two cloud regions. What are the main DevOps challenges and how would you handle them?
- Your DORA metrics show a 30% change failure rate and long recovery times. How would you bring them down without slowing teams down?
- Leadership asks you to cut the cloud bill for your Kubernetes platform by 30 percent without hurting reliability. How do you approach it?
- You are the incident commander for a major outage affecting all customers. Describe how you run the incident from first page to postmortem.
- Design CI/CD for a monorepo with 60 services and 200 engineers where full builds take over an hour.
- Design a disaster recovery plan for a platform whose business requires an RTO of 1 hour and an RPO of 5 minutes for the loss of a whole cloud region.
- Several teams will share one Kubernetes cluster. How would you isolate tenants and secure the cluster?