How should a client retry failed calls to another service safely?

What the interviewer wants to hear

• Only retry on transient, retryable errors
• Exponential backoff with jitter
• Limit attempts and use a retry budget
• Idempotency or idempotency keys for writes
• Respect overall deadlines and retry at one layer only

Model answer

I only retry errors that are likely to be transient, such as timeouts, connection resets, 503 or 429, and not errors like 400 that will fail again. Retries use exponential backoff with random jitter so many clients do not retry together and create a thundering herd. I cap attempts, usually two or three, and use a retry budget, for example retries may not exceed 10% of normal traffic, so a failing dependency is not overloaded. Writes must be idempotent, or I send an idempotency key so the server can safely ignore duplicates. I also propagate a deadline so retries stop when the caller no longer cares, and I retry at only one layer of the stack to avoid multiplying load.

More medium Site Reliability Engineer questions

• How would you design alerting for a service so that on-call engineers are not flooded with noisy alerts?
• Walk me through how you would run and write a postmortem after a major outage.
• A Linux server running a service suddenly becomes slow. How do you investigate?
• How do you do capacity planning for a growing service?
• A team is launching a new checkout API and asks you to help define its SLOs. How do you approach it?
• Your Prometheus servers are running out of memory because of too many time series. How do you find the cause and keep cardinality under control?
• A Deployment rollout is stuck because new pods are in CrashLoopBackOff. Walk me through how you would debug it.
• During a major incident, how do you handle communication with customers, support and leadership?
• A big marketing launch is in three weeks. How would you load test the service to know it will cope?