A product manager says each payment event must be processed exactly once. How do you achieve that with real message queues and networks?

What the interviewer wants to hear

• Exactly-once delivery is not possible across networks in general
• Use at-least-once delivery plus idempotent processing for effectively-once
• Deduplication with unique keys or processed-event table in the same transaction
• Transactional outbox and Kafka transactions where useful
• Handle ordering, retries and poison messages

Model answer

Over an unreliable network you cannot guarantee a message is delivered exactly once: if an acknowledgement is lost, the sender must choose between retrying, which risks duplicates, or not, which risks loss. So we aim for effectively-once: at-least-once delivery combined with idempotent processing. Each event carries a unique id. The consumer, in the same database transaction as its business change, inserts that id into a processed_events table with a unique constraint; a duplicate fails the insert and is skipped. On the producer side, the transactional outbox ensures the event is published if and only if the business data was committed. Kafka offers idempotent producers and transactions for read-process-write inside Kafka, but side effects in external systems still need idempotency keys, for example when calling the payment provider. I also plan for ordering per key, retries and a dead-letter queue.

More hard Backend Developer questions

• Design a URL shortener like bit.ly that handles 100 million new links per month and many more redirects. Cover IDs, storage, caching and scale.
• Your single PostgreSQL database is reaching its limits. How would you scale reads and then writes, and what problems does each step bring?
• In a microservices system, placing an order touches the Order, Payment and Inventory services, each with its own database. How do you keep data consistent?
• You are building a B2B SaaS backend used by many companies. How would you design multi-tenancy and make sure one tenant can never see another tenant data?
• Design a notification service that sends email, SMS and push messages for many products, up to tens of millions of messages a day.
• Your company wants to split a large monolith into microservices. How would you approach it, and when would you argue against it?
• Design the backend for a real-time chat app with one-to-one and group chats, online status and message history.
• One slow downstream dependency caused your whole platform to go down. How would you handle the incident, and what would you change afterwards?
• How would you design a ledger for a wallet or payments product so balances are always correct?