Skip to content
RED SAGSkills
hardData Engineer interview question

Design a CDC pipeline that replicates a busy PostgreSQL database into Iceberg or Delta tables, including updates and deletes. What are the tricky parts?

What the interviewer wants to hear

  • Initial snapshot plus streaming changes without gaps or duplicates
  • Debezium reading the WAL into Kafka, keyed by primary key
  • Apply changes with MERGE using the log sequence number to keep order and dedupe
  • Handle deletes, tombstones and schema changes
  • Manage small files and delete files with compaction and maintenance
  • Monitor lag and run reconciliation against the source

Model answer

I would run Debezium against PostgreSQL logical replication. It takes a consistent initial snapshot and then streams WAL changes into Kafka topics keyed by primary key, so changes for one row stay in order. A Spark or Flink job reads micro-batches, keeps only the latest change per key within each batch using the LSN, and MERGEs into an Iceberg or Delta table: updates overwrite, inserts add, and delete events remove rows or set a soft-delete flag. Using the LSN in the merge condition makes replays idempotent and stops an older event from overwriting a newer one. Schema changes flow through a schema registry, and I allow additive changes automatically but alert on breaking ones. Frequent merges create many small files and delete files, so I schedule compaction and snapshot expiry. Finally I monitor replication slot lag, because a stuck slot fills the source disk, and I run periodic count and checksum reconciliation.

Can you answer it under time pressure?

Take a timed hard round and get scored on every answer.

Start mock interview

More hard Data Engineer questions