medium10 pts · first answer countsdata-pipelinesapi-extraction
Missing rows when paging an API
A job pages through /orders?offset=0&limit=100, then offset=100 and so on, sorted by created_at. New orders arrive during the run and some orders are skipped or repeated. What fixes it?
Discussion 0
Join free to post in the discussion.
Related questions
- ETL vs ELTData Engineer
- Idempotent pipelinesData Engineer
- Batch or streaming for a daily reportData Engineer
- Incremental logic for a tiny tableData Engineer
- What SCD Type 1 doesData Engineer
- Keeping the old customer segmentData Engineer
- Why a dimension has its own keyData Engineer
- The bronze layerData Engineer