Design an event-driven order processing system on a public cloud that must handle traffic spikes and never lose or double-charge an order.
What the interviewer wants to hear
- Durable queue or stream to buffer spikes between producers and consumers
- At-least-once delivery means idempotent consumers with idempotency keys
- Transactional outbox to publish events reliably with database writes
- Dead-letter queues, retries with backoff, and alerts on DLQ depth
- Ordering only where needed, using partition or message group keys
- Saga or step functions for multi-step workflows and compensation
Model answer
The API validates the order, writes it to the database and records an event in an outbox table in the same transaction, so we never save an order without publishing its event. A relay or change data capture pushes outbox events to a queue or stream such as SQS, EventBridge or Kafka, which absorbs spikes while consumers scale on queue depth. Delivery is at-least-once, so every consumer is idempotent: payment calls use the order ID as an idempotency key, and we store processed message IDs. Failures retry with exponential backoff, and messages that keep failing go to a dead-letter queue with an alarm and a redrive process. I only enforce ordering where it matters, by keying on order ID. The multi-step flow of payment, inventory and shipping runs as a saga, for example in Step Functions, with compensating actions like refunds.
Can you answer it under time pressure?
Take a timed hard round and get scored on every answer.
More hard Cloud Engineer questions
- Design a multi-region disaster recovery setup for a payments API with an RPO of under 1 minute and an RTO of 15 minutes. What trade-offs do you consider?
- How would you design a multi-account cloud landing zone for a company with 20 product teams?
- Leadership asks you to reduce the cloud bill by 30 percent without hurting reliability. How do you approach it?
- How would you migrate a large production database to a managed cloud database with near-zero downtime?
- A company needs reliable connectivity between its data center and the cloud. Compare site-to-site VPN and a dedicated connection like AWS Direct Connect, and describe a resilient design.
- Design a shared Kubernetes platform used by 30 product teams. How do you handle isolation, security, cost and upgrades?
- You find out that long-lived AWS access keys were pushed to a public GitHub repository an hour ago. How do you respond?
- Design observability and alerting for a customer-facing API so on-call gets paged for real problems but not for noise.
- Your company has hundreds of Terraform configurations owned by many teams. How do you structure modules, state and governance so it scales safely?