Data Engineer
Data pipelines and data quality questions
132 questions on Data pipelines and data quality for data engineers, easiest first.
Easy 32
- ETL vs ELTeasy
- Idempotent pipelineseasy
- Built-in dbt testseasy
- Batch or streaming for a daily reporteasy
- Incremental logic for a tiny tableeasy
- What SCD Type 1 doeseasy
- Keeping the old customer segmenteasy
- Why a dimension has its own keyeasy
- The bronze layereasy
- Why keep the raw fileseasy
- File format for an analytics lakeeasy
- Partitioning an events tableeasy
- Loading 10 million rowseasy
- Keeping track of deleted rowseasy
- Getting effectively-once resultseasy
- Remembering where the last load stoppedeasy
- Append or upsert for click eventseasy
- What reverse ETL meanseasy
- Which quality dimension is brokeneasy
- Checking a status columneasy
- Guarding the grain of a tableeasy
- Correct numbers that arrive lateeasy
- Orders pointing to missing customerseasy
- First check after a migrationeasy
- A row that says "order_id"easy
- Codes losing their leading zeroseasy
- Data type for amounts in rupeeseasy
- One time zone for stored timeseasy
- Arabic names turning into ????easy
- Blank values that pass a not-null testeasy
- Tools for data testseasy
- Why give a schema for JSONeasy
Medium 59
- Incremental load with updated_atmedium
- Upserting into a target tablemedium
- Table state after a MERGEmedium
- Log-based CDC advantagesmedium
- Duplicates from at-least-once deliverymedium
- Surprise runs from catchupmedium
- Filtering new rows in dbtmedium
- Detecting stale source datamedium
- What a data contract doesmedium
- Rerunning a half-failed loadmedium
- A row-count check that failsmedium
- Choosing the rows for a daily extractmedium
- Selecting current rows from SCD2medium
- Joining facts to the right SCD2 versionmedium
- Keys that look equal but do not joinmedium
- Events that arrive days latemedium
- Rerunning one day safelymedium
- Write, audit, publishmedium
- Dashboards empty during a reloadmedium
- Keeping one row per ordermedium
- Deleted rows still in the warehousemedium
- Starting CDC on an existing tablemedium
- Applying delete events in a MERGEmedium
- Keeping changes to one row in ordermedium
- When to commit Kafka offsetsmedium
- Rerunning COPY on the same foldermedium
- Missing rows when paging an APImedium
- An API that returns 429medium
- Duplicates despite a primary keymedium
- Which date to partition bymedium
- Daily snapshots versus SCD2 rowsmedium
- Planning a two-year backfillmedium
- 20 bad rows out of 2 millionmedium
- Is today's volume an anomalymedium
- Freshness lag against an SLOmedium
- Warning before failing on stale sourcesmedium
- A column that is sometimes nullmedium
- Row counts match but money does notmedium
- Is 03/04/2026 March or Aprilmedium
- Addresses breaking the columnsmedium
- Catching bad JSON at the doormedium
- A new payment method breaks a testmedium
- Which dashboards a bad table affectsmedium
- Joining on email without exposing itmedium
- Phone numbers for some users onlymedium
- Queries slow on a streaming tablemedium
- One 40 GB gzip CSVmedium
- A file processed twicemedium
- One bad message blocks a streammedium
- A one-off business rule testmedium
- The same check on many columnsmedium
- Tolerating a few bad rowsmedium
- Seeing which rows failed a testmedium
- Applying CDC to a lake tablemedium
- Reading only changed rows downstreammedium
- How Debezium captures changesmedium
- How the outbox pattern worksmedium
- Malformed rows in a JSON feedmedium
- Delta MERGE for new and changed rowsmedium
Hard 41
- Applying CDC eventshard
- Upstream renamed a columnhard
- Rows missed by a strict watermarkhard
- Overwrite deleted the whole tablehard
- Doubled rows after a retryhard
- Two runs overwriting one temp filehard
- Postgres disk filling after CDC outagehard
- An older update overwrites a newer onehard
- A row missed despite >= on updated_athard
- MERGE fails on duplicate source rowshard
- A MERGE that scans 5 years of datahard
- Order saved but event never senthard
- Exactly-once inside Kafkahard
- A compacted CDC topichard
- Backfill and daily load collidehard
- Rows missed because of clock skewhard
- Changing a field type in a topichard
- Deleting one person from a data lakehard
- Retries leaving extra fileshard
- Revenue inflated by repeated eventshard
- GST totals off by a few rupeeshard
- Stopping bad data reaching the CFOhard
- Too many data alertshard
- No FX rate for Saturday ordershard
- An order on the wrong IST dayhard
- Average order value up 100 timeshard
- Bank and app never agree on a dayhard
- Testing that a job is safe to rerunhard
- Proving a rewrite changed nothinghard
- Adding consumers to cut laghard
- Birth dates of 1900-01-01hard
- Duplicates that slip past a stream dedupehard
- Consumer down longer than retentionhard
- A new CSV column shifts the datahard
- Revenue tripled after a joinhard
- A Snowflake stream that keeps returning rowshard
- Stopping a column type changehard
- Testing logic on fixed inputshard
- Checking data before readers see ithard
- Enriching events with a changing tablehard
- Speculation and a database sinkhard