How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?

What the interviewer wants to hear

• Readiness probes, rolling update settings, multiple replicas spread across zones
• Graceful shutdown with preStop and terminationGracePeriod, connection draining
• PodDisruptionBudgets and correct resource requests
• Progressive delivery: canary with automated analysis on SLIs
• Fast rollback and backward-compatible changes such as schema migrations

Model answer

I run several replicas spread across zones with topology spread constraints and a PodDisruptionBudget, with realistic resource requests. Rolling updates use maxUnavailable 0 and a small maxSurge, and readiness probes only pass when the app can really serve. For shutdown, the app handles SIGTERM, stops accepting new work and finishes in-flight requests; a short preStop sleep gives load balancers time to remove the pod, and terminationGracePeriodSeconds is long enough. For risk control I use progressive delivery, for example Argo Rollouts, sending 5% of traffic to a canary and automatically comparing error rate and latency against the baseline before promoting. Rollback must be one command or automatic. Changes, especially database migrations, are backward compatible, using expand and contract, so old and new versions can run side by side.

More hard Site Reliability Engineer questions

• What causes cascading failures in distributed systems, and how do you design to prevent them?
• Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
• How would you introduce chaos engineering to an organisation that has never done it?
• Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
• Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
• Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
• After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
• Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
• You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?