How would you introduce chaos engineering to an organisation that has never done it?

What the interviewer wants to hear

• Start with prerequisites: good observability, SLOs, incident process
• Game days and experiments in staging first
• Hypothesis, steady state, small blast radius, abort conditions
• Gradually move to production with stakeholder buy-in
• Turn findings into fixes and automated recurring experiments

Model answer

I would first make sure the basics exist: good monitoring, SLOs and a working incident process, otherwise we cannot tell whether an experiment hurts users. Then I would start with game days in staging, for example killing a pod, adding latency to a dependency or failing over a database, with the owning team watching. Every experiment follows the same steps: define steady state using SLIs, write a hypothesis such as "losing one zone keeps errors under the SLO", limit the blast radius and agree clear abort conditions. Results go into tickets and are fixed like postmortem items. Once teams trust the process, we move carefully to production during working hours with small scope, then automate regular experiments in CI or with tools like Chaos Mesh or AWS FIS.

More hard Site Reliability Engineer questions

• What causes cascading failures in distributed systems, and how do you design to prevent them?
• Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
• How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
• Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
• Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
• Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?
• After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
• Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
• You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?