hard20 pts · first answer countsprometheusobservabilityon-call

Alert silent when the service dies

The alert below never fired when the payments service crashed completely and stopped being scraped. Why, and what is the fix?

- alert: PaymentsHighErrorRate
  expr: |
    sum(rate(http_requests_total{job="payments",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{job="payments"}[5m])) > 0.05
  for: 5m

A.The for: 5m clause keeps resetting while errors stay above 5 percent, so it waits forever.B.Prometheus does not allow division between two sum() aggregations inside an alerting rule.C.With no series the expression is empty, not high, so nothing fires; add an up == 0 or absent() alert.D.Regex label matchers such as code=~"5.." are not supported in alerting rules, only in queries.

Discussion 0

Join free to post in the discussion.

Alert silent when the service dies

Discussion 0

Related questions