You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?
What the interviewer wants to hear
- Gather data on incidents, causes, customer impact and toil
- Introduce SLOs for critical user journeys to define what good looks like
- Prioritise the biggest recurring causes and quick wins like rollback and alerting
- Track outcome metrics such as SLO attainment, incident count, MTTR and toil
- Build shared ownership with development teams through error budgets
Model answer
I start by gathering facts: the last six months of incidents, their causes, customer impact and how they were detected, plus where on-call time and toil go. Usually a few patterns, such as risky deploys, missing alerts or one fragile dependency, cause most of the pain. I then define SLOs for the most important user journeys with product teams, so we agree on what good looks like and have error budgets to guide trade-offs. Early work targets the biggest recurring causes and quick wins: reliable rollback, canary releases, alerts on user symptoms and runbooks. I make sure postmortem action items are tracked to completion. To show progress I report outcome metrics over time: SLO attainment, number and severity of incidents, time to detect and restore, pages per shift and toil hours. Sharing these with leadership and dev teams builds shared ownership of reliability.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Site Reliability Engineer questions
- What causes cascading failures in distributed systems, and how do you design to prevent them?
- Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
- How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
- How would you introduce chaos engineering to an organisation that has never done it?
- Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
- Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
- Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
- After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
- Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?