medium10 pts · first answer countssparkspark-tuning

An iterative loop that gets slower

A job updates a DataFrame in a loop 100 times (df = df.join(...).withColumn(...)). Each iteration gets slower and planning takes minutes. What is a standard fix?

A.Raise spark.sql.shuffle.partitions after each loopB.Checkpoint the DataFrame every few iterations to cut the lineageC.Call df.count() at the end of each loopD.Switch the joins to use a Python UDF

Discussion 0

Join free to post in the discussion.

An iterative loop that gets slower

Discussion 0

Related questions