hard20 pts · first answer countssparkperformance

One task takes forever

A Spark join on customer_id finishes 199 of 200 tasks in 2 minutes, but one task runs for an hour. One customer_id has 40% of all rows. What is the best fix?

A.Add more executors so the slow task gets more memoryB.Increase spark.sql.shuffle.partitions to 2000C.Salt the hot key (add a random suffix and replicate the other side) or enable AQE skew join handlingD.Convert the DataFrame to an RDD

Exactly-once end to end →