Site Reliability Engineer interview questions and answers

An SLI, or service level indicator, is a measurement of how the service behaves for users, for example the ratio of successful HTTP requests to total requests, or the fraction of requests faster than 300 ms. An SLO, or service level objective, is the target we set for that SLI over a window, such as 99.9% of requests succeeding over 30 days. An SLA, or service level agreement, is a contract with customers that includes consequences like refunds if we miss it, for example 99.5% monthly availability. We keep the SLO stricter than the SLA so we notice and fix problems before we break a contract. The SLO also defines the error budget, which is the amount of unreliability we can accept.

The error budget is the amount of failure allowed by the SLO. If the SLO is 99.9% success over 30 days, the budget is 0.1% of requests, or about 43 minutes of full downtime. Because users cannot tell the difference between 99.9% and 100%, we spend that budget on useful risk, such as shipping features and running experiments. The team agrees an error budget policy in advance: while budget remains, releases continue normally; if the budget is used up, we slow or freeze risky launches and spend time on reliability work like fixing the top causes of errors. This turns arguments between developers and operations into a data-driven decision both sides already agreed to.

The four golden signals are latency, traffic, errors and saturation. Latency is how long requests take, and I track percentiles like p50 and p99, keeping successful and failed requests separate because fast errors can make latency look good. Traffic is the demand on the system, such as requests per second. Errors are the rate of failed requests, including implicit failures like a 200 response with wrong content or a response that breaks the SLO. Saturation is how full the service is, for example CPU, memory, queue length or connection pool use. They are useful because they focus on what users experience and give a small, consistent starting point for dashboards and alerts across every service.

First I acknowledge the page so others know someone is on it. Then I check how bad the impact is: which endpoints, regions or customers are affected and how fast we are burning the error budget. I open the runbook for that alert and look at dashboards for the golden signals. Very often the cause is a recent change, so I check deploys, config changes and feature flags in the last hour. My priority is to mitigate, not to find the perfect root cause, so if a recent release lines up with the errors I roll it back or turn off the flag. If the impact is large or I am stuck, I declare an incident, pull in help, and post regular status updates while keeping notes for the postmortem.

Toil is operational work that is manual, repetitive, can be automated, has no lasting value and grows as the service grows, such as restarting a stuck job every night or creating user accounts by hand. Some toil is normal, but too much burns people out and stops engineering progress, so many SRE teams cap it at around 50% of their time. To reduce it, I first measure it, for example by tagging tickets and pages. Then I rank tasks by frequency and pain and automate the top ones, or better, fix the root cause so the task disappears. I also build self-service tools so other teams do not need to file tickets for routine requests.

Monitoring is watching signals we already know matter, like error rate or disk usage, and alerting when they cross a threshold. Observability is a property of the system: how well we can understand new, unexpected problems from the telemetry it emits, without shipping new code. The three main signals work together. Metrics are numeric time series that are cheap to store and great for dashboards, trends and alerts, but they lose detail. Logs are detailed records of individual events, useful for seeing the exact error and context for one request. Traces follow a single request across services as a tree of spans, showing where time was spent and which hop failed. In practice I start from a metric alert, jump to traces for slow or failed requests, and then read the logs linked by trace ID.

Clicking in a console is fast once but hard to repeat, review or undo. With Terraform the desired infrastructure lives in version control, so every change goes through a pull request where teammates can review the code and the terraform plan output, which shows exactly what will be created, changed or destroyed. That catches mistakes like an accidental delete before they reach production. Git history becomes an audit trail and makes rollback simpler. Because the same modules build staging and production, environments stay consistent, and we can rebuild a region quickly after a disaster. Terraform also detects drift when someone changes a resource by hand. We keep the state in a remote backend with locking so that two people cannot apply at the same time.

In a blue-green deployment I run two complete environments. Blue serves users while green gets the new version and is tested; then I switch all traffic to green at the router. Rollback is instant because I just switch back, but it needs double capacity during the release and every user moves at once. In a canary deployment I send a small share of real traffic, say 1 to 5 percent, to the new version and compare its error rate and latency with the old one. If it looks healthy I increase the share in steps; if not, only a few users were affected. I choose canary for most services with enough traffic to compare, and blue-green when I need an all-or-nothing switch or when traffic is too low for a canary to show anything. Both need backward-compatible database changes.

I would design it to answer one question first: are users OK right now? The top row shows SLO status and error budget remaining, then request rate, error rate and latency as p50, p95 and p99, not averages. The next row shows saturation: CPU, memory, connection pool usage and queue depth, because these explain why latency rises. Then a row for key dependencies, such as database and downstream service latency and errors. I add deploy annotations so it is easy to see whether a change lines up with a spike. Variables let people pick the environment, region and service, so one dashboard works everywhere. I keep it short and link to more detailed dashboards and to the runbook, because an on-call engineer at 3 a.m. needs a clear picture, not fifty panels.

A healthy rotation has enough people that each person is on call only every few weeks, with a secondary and a clear escalation policy so nobody is alone. Page volume should be low, a common target is no more than a couple of incidents per shift, and every page must be urgent and actionable; everything else becomes a ticket. Each alert links to a runbook and a dashboard so the engineer knows what to do. At the start of each shift there is a short handoff about ongoing issues. After a heavy night people get time to recover, and on-call work is recognised or paid. Finally, the team reviews pages every week, deletes or tunes noisy alerts and fixes recurring causes. Without that loop alert fatigue and burnout grow.

I start from the SLOs and page only on user-visible symptoms, such as error-budget burn or high latency, not on causes like CPU on one host. I use multi-window, multi-burn-rate alerts: a fast burn, for example 14.4x over one hour confirmed over five minutes, pages someone, while a slow burn over days creates a ticket. Every paging alert must be urgent, actionable and link to a runbook; if nobody needs to act right now, it is a ticket or a dashboard panel. In Alertmanager I group and deduplicate related alerts and route them to the owning team. Finally, I review alerts every few weeks, deleting or tuning any that fired without needing action.

I start soon after the incident while memories are fresh, and I keep it blameless: we ask how the system let a reasonable person make that choice, not who made a mistake. The document includes a summary, user impact and error budget used, and a detailed timeline from the first trigger through detection, mitigation and full recovery. Then we analyse contributing factors, since outages usually have several, such as a risky change, a missing alert and a slow rollback process. We list what went well and where we got lucky. Each action item has an owner, priority and due date and is tracked in our ticket system. Finally, we review it in a meeting and share it widely so other teams learn too.

I only retry errors that are likely to be transient, such as timeouts, connection resets, 503 or 429, and not errors like 400 that will fail again. Retries use exponential backoff with random jitter so many clients do not retry together and create a thundering herd. I cap attempts, usually two or three, and use a retry budget, for example retries may not exceed 10% of normal traffic, so a failing dependency is not overloaded. Writes must be idempotent, or I send an idempotency key so the server can safely ignore duplicates. I also propagate a deadline so retries stop when the caller no longer cares, and I retry at only one layer of the stack to avoid multiplying load.

I follow the USE method: for each resource I check utilization, saturation and errors. I start with uptime for load average, then top or htop to see CPU use and which processes are busy. vmstat 1 shows run queue length, swapping and I/O wait, and free shows memory pressure. iostat -x shows disk utilization and latency, and ss or sar show network connections and errors. dmesg and journalctl can reveal OOM kills, disk errors or throttling. I also check whether something changed recently, such as a deploy or a cron job. If the cause is still unclear, I use perf, flame graphs or eBPF tools like biolatency and offcputime to see where time is really going.

I start with demand: historical traffic trends plus known events such as sales, launches or festival seasons, and I plan for peak, not average. Then I measure supply by load testing to find how much traffic one instance can handle while still meeting the latency SLO, and which resource becomes the bottleneck first, often a database or a downstream service. From that I calculate instances needed at peak and add headroom, plus N+1 or N+2 so we survive losing a zone. I consider lead times for things like quotas, reserved capacity or database scaling, and I balance cost. Autoscaling helps with daily patterns but has limits like warm-up time, so I review the plan every quarter.

I start with the user journeys: adding to cart, entering payment details and placing the order. For each, I ask what users notice when it breaks. That gives SLIs such as the share of checkout requests that succeed, and the share that complete within, say, 500 milliseconds, measured at the load balancer rather than inside the app. Next I look at a few weeks of baseline data and talk with product about how much unreliability the business can accept, then set achievable targets like 99.9% availability over 28 days. A target above what dependencies offer is not realistic. We agree on an error budget policy in writing: what happens when the budget is spent. Then I build multi-window burn-rate alerts and a dashboard, and we review the SLOs every quarter to see if they still match user happiness.

First I find what is growing. The Prometheus TSDB status page lists the metrics and label names with the most series, and a query like topk(10, count by (__name__)({__name__=~".+"})) confirms it. Usually it is one metric with an unbounded label, such as user ID, request ID, full URL path or pod names that change on every deploy. I work with the owning team to remove or bucket that label, for example using a route template instead of the raw path. In the meantime I can drop the label or the metric with metric_relabel_configs. To stop it happening again I set sample_limit per scrape job, add cardinality alerts, and review new instrumentation in code review. Per-request detail belongs in logs or traces, and recording rules can keep useful aggregates.

If users are affected I first make sure old pods are still serving; with maxUnavailable set correctly they are, otherwise I run kubectl rollout undo. Then I run kubectl describe pod on a failing pod and read the last state, exit code and events. Exit code 137 with OOMKilled points to memory limits; exit code 1 usually means the app failed at start-up. Next I run kubectl logs with --previous to see the output of the crashed instance, which often shows a missing environment variable, a bad config value or a failed connection to a dependency. I compare what changed in this release: image, ConfigMaps, Secrets, probes and resources. A liveness probe that is too aggressive can also kill a slow-starting app, so a startup probe may be needed. If it is still unclear, I run the image locally with the same config.

I make sure the incident commander assigns a communications lead, so the people fixing the problem are not interrupted by questions. The comms lead posts a first update quickly, within about 15 minutes, on the status page and the internal incident channel, even if it only says we are investigating elevated errors in checkout. Updates focus on user impact, what we are doing and when the next update will come, for example every 30 minutes, and we keep that promise even when there is nothing new. We avoid guessing about root cause or blame. Support gets a short message they can share with customers, and leadership gets a summary of impact and risk. When it is resolved we post a clear resolution notice, and later share the postmortem summary with the people who were affected.

I start by estimating peak load with marketing and past data, for example three times the normal peak, and by recording the real mix of endpoints. I write k6 scripts that follow that mix, using an arrival-rate executor so the test keeps sending requests even when the system slows down, which avoids hiding tail latency. I run them in a production-like environment with realistic data sizes and dependencies, or carefully in production at quiet times. I ramp up in steps and set thresholds from the SLOs, such as p95 under 300 milliseconds and errors under 0.1%, while watching CPU, memory, connection pools and database load. I push past the target to find the breaking point and the first bottleneck, fix it, and repeat. Finally I plan headroom, autoscaling limits and a degradation plan for launch day.

A cascading failure is a positive feedback loop: one part gets overloaded, slows down or fails, and its load moves to other parts, which then fail too. Common triggers are retry storms, a lost zone shifting traffic, slow dependencies that tie up threads and connection pools, and unbounded queues. To prevent it I set timeouts on every call and propagate deadlines so work stops when the caller has given up. I limit retries with backoff, jitter and retry budgets. Servers protect themselves with bounded queues, concurrency limits and load shedding by priority. Circuit breakers stop calls to unhealthy dependencies, and graceful degradation serves simpler or cached results. Bulkheads isolate resources per dependency or tenant, and enough headroom means losing one zone does not overload the rest. I test all of this with load tests and chaos experiments.

With fan-out, the slowest backend decides the response time, so if each backend is slow 1% of the time and we call 100 of them, most requests hit a slow one. First I look for the sources of variance on backends: garbage collection pauses, CPU throttling, noisy neighbours, background jobs like compaction, and queueing at high utilization. Then I add tail-tolerant techniques. Hedged requests send a second copy to another replica if the first has not replied by around the p95, which cuts the tail for a small extra cost; tied requests let replicas cancel each other. Load balancers like least-request or power of two choices avoid slow replicas. Where the product allows, I return partial results after a per-backend timeout, and I reduce fan-out by better sharding or caching.

I run several replicas spread across zones with topology spread constraints and a PodDisruptionBudget, with realistic resource requests. Rolling updates use maxUnavailable 0 and a small maxSurge, and readiness probes only pass when the app can really serve. For shutdown, the app handles SIGTERM, stops accepting new work and finishes in-flight requests; a short preStop sleep gives load balancers time to remove the pod, and terminationGracePeriodSeconds is long enough. For risk control I use progressive delivery, for example Argo Rollouts, sending 5% of traffic to a canary and automatically comparing error rate and latency against the baseline before promoting. Rollback must be one command or automatic. Changes, especially database migrations, are backward compatible, using expand and contract, so old and new versions can run side by side.

I would first make sure the basics exist: good monitoring, SLOs and a working incident process, otherwise we cannot tell whether an experiment hurts users. Then I would start with game days in staging, for example killing a pod, adding latency to a dependency or failing over a database, with the owning team watching. Every experiment follows the same steps: define steady state using SLIs, write a hypothesis such as "losing one zone keeps errors under the SLO", limit the blast radius and agree clear abort conditions. Results go into tickets and are fixed like postmortem items. Once teams trust the process, we move carefully to production during working hours with small scope, then automate regular experiments in CI or with tools like Chaos Mesh or AWS FIS.

If all five are hard dependencies on the critical path, their availability multiplies: 0.999 to the power 5 is about 99.5%, before counting our own failures. So promising 99.99% would be impossible without changing the design. I would first ask whether users really need 99.99%, because each extra nine costs much more engineering and money. If it is needed, I would remove dependencies from the critical path: cache responses, serve stale data, move work to asynchronous queues, degrade gracefully by hiding non-essential features, and add redundancy such as calling replicas in different regions. I would also check real measured availability of each dependency, since it may be better than its SLO, and agree an SLO that matches what the architecture can actually deliver.

I first agree on RTO and RPO with the business, because they drive cost. Active-passive is cheaper and simpler but failover is slower and the standby may be untested; active-active gives fast failover but needs data that can be written in more than one region. For data, synchronous replication avoids loss but adds latency on every write; asynchronous replication is fast but loses recent writes on failover, so some data may need conflict handling or a single home region. Traffic steering uses health-checked global load balancing or anycast, because DNS-only failover is limited by client caching. Each region needs enough headroom to take the others traffic, or autoscaling with reserved quota. I look for hidden single-region dependencies such as the identity provider, CI, secrets or a control plane. Finally, we run regular failover drills, because an untested failover plan usually fails.

I start with data, breaking the bill down by signal, team and service, because usually a few sources cause most of the cost. For logs, I drop debug output in production, stop logging every successful health check, sample routine success logs, and move older logs to cheaper storage with shorter hot retention. For metrics, I find high-cardinality series and metrics no dashboard or alert uses, then remove or aggregate them with relabeling and recording rules. For traces, I switch from keeping a fixed share of everything to tail-based sampling in the OpenTelemetry Collector, keeping all errors and slow traces plus a small sample of normal ones. Doing this at the Collector means no app changes. Then I give each team a visible cost and a budget, and add limits such as sample_limit so growth is caught early. I check that incident debugging still works after each change.

This is a metastable failure: the trigger is gone, but retries and work for requests nobody is waiting for keep the system overloaded. Adding capacity often does not help because retry traffic grows with it. To recover, I break the feedback loop: shed load at the edge, turn down or disable client retries through config or feature flags, and, if needed, block a share of traffic and let it back in gradually while queues drain. Servers should drop requests whose deadline has already passed instead of processing them. To prevent it, I use retry budgets that cap retries to a small share of normal traffic, exponential backoff with jitter, circuit breakers on dependencies, and retries at only one layer. Deadline propagation and admission control keep useless work out. Finally I load test recovery itself, not only peak load, to prove the system returns to normal after a spike.

I treat configuration exactly like code. Every change lives in version control, goes through review, and is checked by automated validation such as schema checks, linting and a dry run against real services in staging. Rollout is progressive: a canary set of hosts first, then one zone, then one region, and only then everywhere, with bake time at each step. At every stage automatic checks compare SLIs like error rate and latency with the baseline and stop or reverse the rollout if they get worse. Rollback must be one click or automatic, and services should keep a last-known-good config so that a bad or empty config file is rejected rather than applied. Global switches that change the whole fleet at once are avoided, and emergency changes still follow a short version of the process. Feature flags follow the same rules, because they are configuration too.

I start by gathering facts: the last six months of incidents, their causes, customer impact and how they were detected, plus where on-call time and toil go. Usually a few patterns, such as risky deploys, missing alerts or one fragile dependency, cause most of the pain. I then define SLOs for the most important user journeys with product teams, so we agree on what good looks like and have error budgets to guide trade-offs. Early work targets the biggest recurring causes and quick wins: reliable rollback, canary releases, alerts on user symptoms and runbooks. I make sure postmortem action items are tracked to completion. To show progress I report outcome metrics over time: SLO attainment, number and severity of incidents, time to detect and restore, pages per shift and toil hours. Sharing these with leadership and dev teams builds shared ownership of reliability.