Data Engineer question bank
100 questions. Filter by level and skill, track what you've solved, and see how many people get each one right.
- ETL vs ELT
- Why Parquet for analytics
- Fact and dimension tables
- Idempotent pipelines
- What is a DAG
- Purpose of a primary key
- When streaming is needed
- Partitioning a big table
- Row count of a LEFT JOIN
- Filtering with = NULL
- COUNT(*) vs COUNT(column)
- Filtering groups with HAVING
- Adding a number to NULL
- Star vs snowflake schema
- Grain of a fact table
- Why use surrogate keys
- SCD type that overwrites
- Goal of normalisation
- Row store or column store
- Weaknesses of CSV
- Where Avro fits
- Kafka consumer groups
- What a Kafka offset is
- Retrying flaky Airflow tasks
- What an Airflow sensor does
- Reading a cron schedule
- When Spark actually runs
- What a Spark partition is
- What a dbt model is
- Built-in dbt tests
- Data lake vs warehouse
- How object storage works
- Protecting PII in analytics
- A CSV bigger than memory
- Tracking customer address history
- Incremental load with updated_at
- What causes a shuffle
- Joining with a small lookup table
- Ordering in Kafka
- Upserting into a target table
- Why use ref() in dbt
- Handling late events
- Keeping the latest row per key
- RANK and DENSE_RANK with ties
- NOT IN with a NULL in the list
- Day-over-day change with LAG
- WHERE filter after a LEFT JOIN
- Counting orders for every customer
- Table state after a MERGE
- Revenue doubled after a join
- Employees above their team average
- Keeping just the previous value
- Data Vault building blocks
- Log-based CDC advantages
- Duplicates from at-least-once delivery
- Surprise runs from catchup
- When a daily Airflow run starts
- When to cache in Spark
- repartition vs coalesce
- Eight consumers, six partitions
- How Parquet skips data
- Undoing a bad load
- LIMIT and BigQuery cost
- Idle Snowflake warehouse cost
- Filtering new rows in dbt
- Detecting stale source data
- What a data contract does
- Rerunning a half-failed load
- One task takes forever
- Exactly-once end to end
- Millions of tiny files
- Safe backfills
- Applying CDC events
- Cutting warehouse scan cost
- Upstream renamed a column
- Why table formats exist
- Running total with tied dates
- LAST_VALUE returns the current row
- Longest login streak
- FULL OUTER JOIN row count
- GROUP BY with NULL keys
- Recursive CTE stopping point
- Double counting in an SCD2 join
- Which late event is kept
- Committing offsets too early
- Adding partitions to a keyed topic
- A slow Python UDF in PySpark
- Distinct users on billions of rows
- Iceberg hidden partitioning
- VACUUM and time travel
- Two writers, one lakehouse table
- Copy-on-write vs merge-on-read
- Clustering for two filter columns
- Choosing a Redshift DISTKEY
- Rows missed by a strict watermark
- New column in a dbt incremental model
- Hashing emails is not anonymising
- Overwrite deleted the whole table
- Which day does an event belong to
- Sensors blocking every worker