Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?

What the interviewer wants to hear

• Treat config as code: version control, review and validation
• Validate with schema checks and pre-production tests before rollout
• Roll out progressively by canary, zone and region with automatic health checks
• Make rollback fast and automatic when SLIs degrade
• Keep last-known-good config and avoid global changes in one step

Model answer

I treat configuration exactly like code. Every change lives in version control, goes through review, and is checked by automated validation such as schema checks, linting and a dry run against real services in staging. Rollout is progressive: a canary set of hosts first, then one zone, then one region, and only then everywhere, with bake time at each step. At every stage automatic checks compare SLIs like error rate and latency with the baseline and stop or reverse the rollout if they get worse. Rollback must be one click or automatic, and services should keep a last-known-good config so that a bad or empty config file is rejected rather than applied. Global switches that change the whole fleet at once are avoided, and emergency changes still follow a short version of the process. Feature flags follow the same rules, because they are configuration too.

More hard Site Reliability Engineer questions

• What causes cascading failures in distributed systems, and how do you design to prevent them?
• Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
• How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
• How would you introduce chaos engineering to an organisation that has never done it?
• Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
• Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
• Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
• After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
• You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?