A Spark job that used to take 20 minutes now takes 2 hours. How do you investigate?
What the interviewer wants to hear
- Check what changed: data volume, code, configs, cluster
- Use the Spark UI to find slow stages and task time distribution
- Look for skew, spills, huge shuffles or too few or too many partitions
- Check the query plan for missing filter pushdown or broadcast joins
- Check input for small files or a changed layout
- Fix the specific bottleneck and measure again
Model answer
I start by asking what changed: input size, a code deploy, Spark version, cluster size or upstream file layout. Then I open the Spark UI and find the stage that got slow. In that stage I look at task durations: if a few tasks run much longer than the median, it is skew on a join or group key, which I fix with AQE skew handling, salting or filtering hot keys. If I see large spill to disk, partitions are too big or memory is short, so I tune shuffle partitions or executor memory. I check the SQL plan with explain to see whether a join stopped broadcasting or a filter is no longer pushed down. I also check the input: thousands of tiny files suddenly appearing would slow file listing and scheduling, so compaction would help. I change one thing at a time and compare the runs.
Can you answer it under time pressure?
Take a timed medium round and get scored on every answer.
More medium Data Engineer questions
- Your source system sometimes sends the same event twice. How do you handle duplicates in your pipeline?
- A dbt model that used to take 5 minutes now takes 2 hours. How do you troubleshoot it?
- A stakeholder asks for a "real-time" dashboard. How do you decide whether to build streaming?
- Your Spark job fails with executor out-of-memory errors. What do you check and change?
- How would you design an incremental load for a large orders table that also receives updates to old orders?
- Your main nightly Airflow DAG failed at 3 a.m. and dashboards are due at 8 a.m. Walk me through how you handle it.
- How would you build a pipeline that maintains an SCD Type 2 customer dimension from a daily source extract?
- When would you choose a lakehouse with Delta Lake or Iceberg over a cloud warehouse like Snowflake or BigQuery, or use both?
- Product engineers keep breaking your pipelines by changing database columns without warning. How would you fix this?