Skip to content
RED SAGSkills
mediumData Engineer interview question

Your Spark job fails with executor out-of-memory errors. What do you check and change?

What the interviewer wants to hear

  • Look at the Spark UI to find the failing stage
  • Check for data skew in the failing tasks
  • Avoid collect() and large driver operations
  • Tune partitions so each is a reasonable size
  • Prefer built-in functions to Python UDFs, filter and select early
  • Adjust memory settings as a last step

Model answer

I open the Spark UI and find which stage and tasks fail. If a few tasks process far more data than others, it is skew, and I fix it with AQE skew join handling or salting. If all tasks are big, partitions are too large, so I increase shuffle partitions or repartition so each partition is roughly 100 to 200 MB. I check for collect() or toPandas() on large data, which overload the driver. I check whether a broadcast join is broadcasting a table that is actually large. I reduce data early by filtering rows and selecting only needed columns before joins, and replace Python UDFs with built-in functions where possible. Caching large DataFrames that are not reused also wastes memory. Only after these fixes would I increase executor memory or memory overhead, because more memory often hides the real problem.

Can you answer it under time pressure?

Take a timed medium round and get scored on every answer.

Start mock interview

More medium Data Engineer questions