Design observability and alerting for a customer-facing API so on-call gets paged for real problems but not for noise.
What the interviewer wants to hear
- Define SLIs such as availability and latency, and SLOs with error budgets
- Page on symptoms using multi-window burn rate alerts
- Cause metrics like CPU or queue depth go to dashboards or tickets
- Golden signals plus traces and structured logs for fast diagnosis
- Every alert has an owner and a runbook, and is reviewed for noise
- Control cardinality and cost of telemetry
Model answer
I start from what users feel. I define SLIs such as the share of requests that succeed and the share served under 300 ms, measured at the load balancer or gateway, and set SLOs like 99.9 percent over 30 days, which gives an error budget. Paging alerts are multi-window burn rate alerts on those SLOs: a fast burn pages within minutes, a slow burn opens a ticket. Cause signals like CPU, memory, queue depth or disk go to dashboards and low-priority alerts, not pages. For diagnosis I collect the golden signals per service, distributed traces with OpenTelemetry, and structured logs with trace IDs. Every paging alert must have an owner and a runbook, and we review alert volume each week and delete or tune noisy ones. I also watch telemetry cost by limiting label cardinality and sampling traces.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Cloud Engineer questions
- Design a multi-region disaster recovery setup for a payments API with an RPO of under 1 minute and an RTO of 15 minutes. What trade-offs do you consider?
- How would you design a multi-account cloud landing zone for a company with 20 product teams?
- Leadership asks you to reduce the cloud bill by 30 percent without hurting reliability. How do you approach it?
- How would you migrate a large production database to a managed cloud database with near-zero downtime?
- A company needs reliable connectivity between its data center and the cloud. Compare site-to-site VPN and a dedicated connection like AWS Direct Connect, and describe a resilient design.
- Design a shared Kubernetes platform used by 30 product teams. How do you handle isolation, security, cost and upgrades?
- You find out that long-lived AWS access keys were pushed to a public GitHub repository an hour ago. How do you respond?
- Design an event-driven order processing system on a public cloud that must handle traffic spikes and never lose or double-charge an order.
- Your company has hundreds of Terraform configurations owned by many teams. How do you structure modules, state and governance so it scales safely?