Design a disaster recovery plan for a platform whose business requires an RTO of 1 hour and an RPO of 5 minutes for the loss of a whole cloud region.
What the interviewer wants to hear
- Define RTO and RPO per system and choose a DR pattern such as warm standby
- Continuous cross-region data replication to meet a 5 minute RPO
- Infrastructure as code and GitOps to recreate compute quickly
- DNS or global load balancer failover and dependency checks
- Backups separate from replication, protected from deletion
- Regular DR drills with measured recovery times and runbooks
Model answer
I start by mapping systems to their real RTO and RPO, because not everything needs the same tier. For a 1 hour RTO and 5 minute RPO on region loss, backup and restore alone is usually too slow, so I would choose warm standby. Databases replicate continuously to the second region, for example cross-region read replicas or a global database, keeping lag well under five minutes, and object storage uses cross-region replication. The standby region runs a small Kubernetes cluster built from the same Terraform code, with Argo CD syncing the same manifests, so scaling it up is fast. Images and secrets are replicated too. Failover is driven by a runbook: promote the replica, scale out, then switch traffic through DNS or a global load balancer. Separate backups, immutable and in another account, protect against corruption that replication would copy. Finally, we run DR drills at least twice a year and measure actual recovery times against the targets.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard DevOps Engineer questions
- You are asked to build an internal platform so 30 product teams can deploy services to Kubernetes on their own. How would you design it?
- How would you secure the software supply chain from source code to a running container in production?
- Your company wants to run a critical service active-active in two cloud regions. What are the main DevOps challenges and how would you handle them?
- Your DORA metrics show a 30% change failure rate and long recovery times. How would you bring them down without slowing teams down?
- How do you plan and carry out a Kubernetes version upgrade for production clusters with minimal risk?
- Leadership asks you to cut the cloud bill for your Kubernetes platform by 30 percent without hurting reliability. How do you approach it?
- You are the incident commander for a major outage affecting all customers. Describe how you run the incident from first page to postmortem.
- Design CI/CD for a monorepo with 60 services and 200 engineers where full builds take over an hour.
- Several teams will share one Kubernetes cluster. How would you isolate tenants and secure the cluster?