How would you design alerting for a service so that on-call engineers are not flooded with noisy alerts?

What the interviewer wants to hear

• Page on symptoms and SLO burn, not on causes
• Every page must be urgent and actionable with a runbook
• Use multi-window burn-rate alerts
• Send non-urgent issues to tickets or dashboards
• Group, deduplicate and route alerts; review alerts regularly

Model answer

I start from the SLOs and page only on user-visible symptoms, such as error-budget burn or high latency, not on causes like CPU on one host. I use multi-window, multi-burn-rate alerts: a fast burn, for example 14.4x over one hour confirmed over five minutes, pages someone, while a slow burn over days creates a ticket. Every paging alert must be urgent, actionable and link to a runbook; if nobody needs to act right now, it is a ticket or a dashboard panel. In Alertmanager I group and deduplicate related alerts and route them to the owning team. Finally, I review alerts every few weeks, deleting or tuning any that fired without needing action.

More medium Site Reliability Engineer questions

• Walk me through how you would run and write a postmortem after a major outage.
• How should a client retry failed calls to another service safely?
• A Linux server running a service suddenly becomes slow. How do you investigate?
• How do you do capacity planning for a growing service?
• A team is launching a new checkout API and asks you to help define its SLOs. How do you approach it?
• Your Prometheus servers are running out of memory because of too many time series. How do you find the cause and keep cardinality under control?
• A Deployment rollout is stuck because new pods are in CrashLoopBackOff. Walk me through how you would debug it.
• During a major incident, how do you handle communication with customers, support and leadership?
• A big marketing launch is in three weeks. How would you load test the service to know it will cope?