hard20 pts · first answer countsdata-pipelinesincremental-loads
Rows missed because of clock skew
updated_at is set by 6 app servers using their own clocks, and one server runs 4 minutes behind. An incremental load with a strict watermark misses some of its rows. What is the most robust fix?
Discussion 0
Join free to post in the discussion.
Related questions
- ETL vs ELTData Engineer
- Idempotent pipelinesData Engineer
- Batch or streaming for a daily reportData Engineer
- Incremental logic for a tiny tableData Engineer
- What SCD Type 1 doesData Engineer
- Keeping the old customer segmentData Engineer
- Why a dimension has its own keyData Engineer
- The bronze layerData Engineer