You are the incident commander for a major outage affecting all customers. Describe how you run the incident from first page to postmortem.
What the interviewer wants to hear
- Declare the incident, set severity and open a dedicated channel
- Assign roles: incident commander, operations lead, communications
- Focus on mitigation first: rollback, failover, feature flag off
- Regular status updates to stakeholders and customers
- Keep a timeline of actions and decisions
- Blameless postmortem with owned, tracked action items
Model answer
First I declare the incident and its severity, open a dedicated channel and video bridge, and page the right owners. I assign roles: I stay incident commander and coordinate rather than debug; an operations lead drives the technical work; and a communications lead posts updates to the status page and internal stakeholders on a fixed schedule, such as every 30 minutes. Our first goal is to stop customer impact, not to find the root cause, so we look at recent changes and use the quickest safe mitigation: roll back, fail over to another region, disable a feature flag or shed load. I keep the team focused, avoid many people changing production at once, and make sure every action is announced and logged in a timeline. After recovery we confirm metrics are healthy. Within a few days we hold a blameless postmortem covering impact, timeline, contributing factors and detection gaps, with action items that have owners and due dates.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard DevOps Engineer questions
- You are asked to build an internal platform so 30 product teams can deploy services to Kubernetes on their own. How would you design it?
- How would you secure the software supply chain from source code to a running container in production?
- Your company wants to run a critical service active-active in two cloud regions. What are the main DevOps challenges and how would you handle them?
- Your DORA metrics show a 30% change failure rate and long recovery times. How would you bring them down without slowing teams down?
- How do you plan and carry out a Kubernetes version upgrade for production clusters with minimal risk?
- Leadership asks you to cut the cloud bill for your Kubernetes platform by 30 percent without hurting reliability. How do you approach it?
- Design CI/CD for a monorepo with 60 services and 200 engineers where full builds take over an hour.
- Design a disaster recovery plan for a platform whose business requires an RTO of 1 hour and an RPO of 5 minutes for the loss of a whole cloud region.
- Several teams will share one Kubernetes cluster. How would you isolate tenants and secure the cluster?