Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
What the interviewer wants to hear
- Explain how fan-out amplifies backend tail latency
- Hedged or backup requests and tied requests
- Find causes of variance such as GC, noisy neighbours, queueing
- Reduce fan-out, use partial results or timeouts per backend
- Load balancing that avoids slow replicas
Model answer
With fan-out, the slowest backend decides the response time, so if each backend is slow 1% of the time and we call 100 of them, most requests hit a slow one. First I look for the sources of variance on backends: garbage collection pauses, CPU throttling, noisy neighbours, background jobs like compaction, and queueing at high utilization. Then I add tail-tolerant techniques. Hedged requests send a second copy to another replica if the first has not replied by around the p95, which cuts the tail for a small extra cost; tied requests let replicas cancel each other. Load balancers like least-request or power of two choices avoid slow replicas. Where the product allows, I return partial results after a per-backend timeout, and I reduce fan-out by better sharding or caching.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Site Reliability Engineer questions
- What causes cascading failures in distributed systems, and how do you design to prevent them?
- How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
- How would you introduce chaos engineering to an organisation that has never done it?
- Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
- Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
- Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
- After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
- Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
- You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?