Your observability bill (metrics, logs and traces) has tripled in a year and leadership wants it cut by 40% without losing the ability to debug incidents. How do you do it?

What the interviewer wants to hear

• Measure cost per signal, team and service to find the biggest sources
• Reduce log volume: drop debug noise, sample success logs, shorter retention tiers
• Control metric cardinality and remove unused metrics
• Use tail-based sampling for traces that keeps errors and slow requests
• Add ownership, budgets and guard rails so costs stay down

Model answer

I start with data, breaking the bill down by signal, team and service, because usually a few sources cause most of the cost. For logs, I drop debug output in production, stop logging every successful health check, sample routine success logs, and move older logs to cheaper storage with shorter hot retention. For metrics, I find high-cardinality series and metrics no dashboard or alert uses, then remove or aggregate them with relabeling and recording rules. For traces, I switch from keeping a fixed share of everything to tail-based sampling in the OpenTelemetry Collector, keeping all errors and slow traces plus a small sample of normal ones. Doing this at the Collector means no app changes. Then I give each team a visible cost and a budget, and add limits such as sample_limit so growth is caught early. I check that incident debugging still works after each change.

More hard Site Reliability Engineer questions

• What causes cascading failures in distributed systems, and how do you design to prevent them?
• Your service fans out to many backends and its p99 latency is much worse than the backends. How would you reduce tail latency?
• How do you make deployments on Kubernetes reliable with zero downtime and safe rollback?
• How would you introduce chaos engineering to an organisation that has never done it?
• Your service depends on five other services, each with a 99.9% availability SLO. You are asked to offer a 99.99% SLO. How do you respond?
• Design a strategy so that a customer-facing service survives the loss of an entire cloud region. What trade-offs do you consider?
• After a brief dependency outage, your service stays overloaded long after the dependency recovered. How do you get out of this state, and how do you design to avoid it?
• Many large outages are caused by configuration changes rather than code. How would you design a safe configuration change process for a large fleet?
• You lead a new SRE team in a company with frequent outages. How do you decide what to work on first, and how do you show the work is paying off?