hard20 pts · first answer countssparkspark-tuningpyspark

Adding 300 columns with withColumn

Code adds 300 derived columns with a for loop that calls df = df.withColumn(...) each time. The job spends minutes before any task starts. What is the better pattern?

A.Cache the DataFrame after every withColumn callB.Use more shuffle partitions for the jobC.Build all expressions and add them in one select()D.Run the loop on the executors with mapPartitions

Discussion 0

Join free to post in the discussion.

Adding 300 columns with withColumn

Discussion 0

Related questions