Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
What the interviewer wants to hear
- Choose active-active or active-passive based on RTO, RPO and cost
- Data replication: synchronous vs asynchronous and the data loss it implies
- Traffic steering with health checks, anycast or DNS and its caching limits
- Capacity headroom so surviving regions can absorb the load
- Regular failover drills and avoiding hidden single-region dependencies
Model answer
I first agree on RTO and RPO with the business, because they drive cost. Active-passive is cheaper and simpler but failover is slower and the standby may be untested; active-active gives fast failover but needs data that can be written in more than one region. For data, synchronous replication avoids loss but adds latency on every write; asynchronous replication is fast but loses recent writes on failover, so some data may need conflict handling or a single home region. Traffic steering uses health-checked global load balancing or anycast, because DNS-only failover is limited by client caching. Each region needs enough headroom to take the others traffic, or autoscaling with reserved quota. I look for hidden single-region dependencies such as the identity provider, CI, secrets or a control plane. Finally, we run regular failover drills, because an untested failover plan usually fails.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Site Reliability Engineer questions
- What causes cascading failures in distributed systems, and how do you design to prevent them?
- Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
- How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
- How would you introduce chaos engineering to an organisation that has never done it?
- Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
- Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
- After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
- Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
- You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?