Explain at-most-once, at-least-once and exactly-once processing, and how you achieve effectively-once results in practice.

Question

RED SAG · Accepted Answer

At-most-once means a message is processed zero or one time: you commit the offset before processing, so a crash loses it. At-least-once commits after processing, so a crash causes a replay and possible duplicates. Exactly-once means each message affects the result once, even with failures. Inside Kafka, idempotent producers and transactions let a consume-transform-produce job commit output and offsets atomically. Engines like Flink and Spark Structured Streaming use checkpoints that store offsets and state together and replay from the last checkpoint. But end-to-end exactly-once also needs the sink to cooperate: either a transactional sink that commits with the checkpoint, like Flink two-phase commit, or an idempotent sink, such as upserting by a unique event ID. In practice, I design for at-least-once delivery plus idempotent writes, which gives effectively-once results and is simpler to reason about.

Explain at-most-once, at-least-once and exactly-once processing, and how you achieve effectively-once results in practice.

What the interviewer wants to hear

Model answer

More hard Data Engineer questions