Your Prometheus servers are running out of memory because of too many time series. How do you find the cause and keep cardinality under control?

What the interviewer wants to hear

• Use TSDB status and queries to find the metrics and labels with most series
• Remove unbounded labels such as user IDs, request IDs and raw paths
• Use relabeling or metric_relabel_configs to drop or aggregate
• Set limits such as sample_limit and review new metrics
• Move high-detail data to logs or traces

Model answer

First I find what is growing. The Prometheus TSDB status page lists the metrics and label names with the most series, and a query like topk(10, count by (__name__)({__name__=~".+"})) confirms it. Usually it is one metric with an unbounded label, such as user ID, request ID, full URL path or pod names that change on every deploy. I work with the owning team to remove or bucket that label, for example using a route template instead of the raw path. In the meantime I can drop the label or the metric with metric_relabel_configs. To stop it happening again I set sample_limit per scrape job, add cardinality alerts, and review new instrumentation in code review. Per-request detail belongs in logs or traces, and recording rules can keep useful aggregates.

More medium Site Reliability Engineer questions

• How would you design alerting for a service so that on-call engineers are not flooded with noisy alerts?
• Walk me through how you would run and write a postmortem after a major outage.
• How should a client retry failed calls to another service safely?
• A Linux server running a service suddenly becomes slow. How do you investigate?
• How do you do capacity planning for a growing service?
• A team is launching a new checkout API and asks you to help define its SLOs. How do you approach it?
• A Deployment rollout is stuck because new pods are in CrashLoopBackOff. Walk me through how you would debug it.
• During a major incident, how do you handle communication with customers, support and leadership?
• A big marketing launch is in three weeks. How would you load test the service to know it will cope?