Product engineers keep breaking your pipelines by changing database columns without warning. How would you fix this?
What the interviewer wants to hear
- Agree on data contracts for key datasets: schema, semantics, owners, SLAs
- Publish events or views as an interface instead of reading raw tables
- Add automated schema checks in the producer CI
- Version breaking changes with a deprecation period
- Detect drift in pipelines and fail fast with clear alerts
- Build the relationship: show impact, make it easy for producers
Model answer
The root problem is that the data team depends on internal tables nobody promised to keep stable. I would pick the most critical datasets and agree a data contract with each owning team: the fields, types, meaning, freshness and who owns it. Ideally producers publish a stable interface, such as outbox events with a schema in a schema registry or a dedicated view, rather than us reading their raw tables. Then I make the contract executable: a check in their CI fails the pull request if a contracted field is removed or changes type, and breaking changes ship as a new version with a deprecation window. On our side, pipelines validate schemas on ingest and fail fast with clear alerts instead of loading bad data. I also show product teams which dashboards and models broke, so they see the impact and treat data as a product.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Data Engineer questions
- Your source system sometimes sends the same event twice. How do you handle duplicates in your pipeline?
- A dbt model that used to take 5 minutes now takes 2 hours. How do you troubleshoot it?
- A stakeholder asks for a "real-time" dashboard. How do you decide whether to build streaming?
- Your Spark job fails with executor out-of-memory errors. What do you check and change?
- How would you design an incremental load for a large orders table that also receives updates to old orders?
- Your main nightly Airflow DAG failed at 3 a.m. and dashboards are due at 8 a.m. Walk me through how you handle it.
- How would you build a pipeline that maintains an SCD Type 2 customer dimension from a daily source extract?
- When would you choose a lakehouse with Delta Lake or Iceberg over a cloud warehouse like Snowflake or BigQuery, or use both?
- A Spark job that used to take 20 minutes now takes 2 hours. How do you investigate?