What causes cascading failures in distributed systems, and how do you design to prevent them?

What the interviewer wants to hear

• Overload causing a positive feedback loop
• Retry storms and resource exhaustion like threads or connections
• Timeouts and deadline propagation
• Load shedding, rate limiting and bounded queues
• Circuit breakers and graceful degradation
• Isolation such as bulkheads and capacity headroom

Model answer

A cascading failure is a positive feedback loop: one part gets overloaded, slows down or fails, and its load moves to other parts, which then fail too. Common triggers are retry storms, a lost zone shifting traffic, slow dependencies that tie up threads and connection pools, and unbounded queues. To prevent it I set timeouts on every call and propagate deadlines so work stops when the caller has given up. I limit retries with backoff, jitter and retry budgets. Servers protect themselves with bounded queues, concurrency limits and load shedding by priority. Circuit breakers stop calls to unhealthy dependencies, and graceful degradation serves simpler or cached results. Bulkheads isolate resources per dependency or tenant, and enough headroom means losing one zone does not overload the rest. I test all of this with load tests and chaos experiments.

More hard Site Reliability Engineer questions

• Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
• How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
• How would you introduce chaos engineering to an organisation that has never done it?
• Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
• Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
• Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
• After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
• Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
• You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?