A stakeholder asks for a "real-time" dashboard. How do you decide whether to build streaming?

What the interviewer wants to hear

• Clarify what latency is actually needed and what decision it drives
• Micro-batch every few minutes is often enough
• Streaming adds cost and complexity: state, late data, operations
• Consider the source capabilities, such as CDC
• Propose an option with clear trade-offs

Model answer

I first ask what real-time means for them and what action they will take with the data. Often "real-time" means updated every 15 minutes, which a micro-batch pipeline can do cheaply. If a human looks at it twice a day, hourly is fine. True streaming with seconds of latency makes sense when a system reacts automatically, like fraud blocking or live inventory. Streaming has real costs: always-on infrastructure, handling late and out-of-order events, state management, exactly-once concerns and harder debugging. I also check the source: can we get CDC or events, or only nightly exports? Then I propose options, for example a 10-minute incremental load now and a streaming design later if the business case is proven, with cost and effort for each. This keeps the solution matched to the real need.

More medium Data Engineer questions

• Your source system sometimes sends the same event twice. How do you handle duplicates in your pipeline?
• A dbt model that used to take 5 minutes now takes 2 hours. How do you troubleshoot it?
• Your Spark job fails with executor out-of-memory errors. What do you check and change?
• How would you design an incremental load for a large orders table that also receives updates to old orders?
• Your main nightly Airflow DAG failed at 3 a.m. and dashboards are due at 8 a.m. Walk me through how you handle it.
• How would you build a pipeline that maintains an SCD Type 2 customer dimension from a daily source extract?
• When would you choose a lakehouse with Delta Lake or Iceberg over a cloud warehouse like Snowflake or BigQuery, or use both?
• A Spark job that used to take 20 minutes now takes 2 hours. How do you investigate?
• Product engineers keep breaking your pipelines by changing database columns without warning. How would you fix this?