After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?

What the interviewer wants to hear

• Recognise a metastable state driven by retries and timed-out work
• Break the loop: shed load, cut retries, rate limit or temporarily block traffic
• Drop work whose deadline has passed and recover gradually
• Use retry budgets, backoff with jitter and circuit breakers
• Use deadline propagation, admission control and load tests of recovery

Model answer

This is a metastable failure: the trigger is gone, but retries and work for requests nobody is waiting for keep the system overloaded. Adding capacity often does not help because retry traffic grows with it. To recover, I break the feedback loop: shed load at the edge, turn down or disable client retries through config or feature flags, and, if needed, block a share of traffic and let it back in gradually while queues drain. Servers should drop requests whose deadline has already passed instead of processing them. To prevent it, I use retry budgets that cap retries to a small share of normal traffic, exponential backoff with jitter, circuit breakers on dependencies, and retries at only one layer. Deadline propagation and admission control keep useless work out. Finally I load test recovery itself, not only peak load, to prove the system returns to normal after a spike.

More hard Site Reliability Engineer questions

• What causes cascading failures in distributed systems, and how do you design to prevent them?
• Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
• How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
• How would you introduce chaos engineering to an organisation that has never done it?
• Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
• Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
• Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
• Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
• You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?