hard20 pts · first answer countssparkspark-partitionsspark-tuning

coalesce(1) made the whole job slow

A job filters and transforms 300 GB, then calls coalesce(1) before writing a single file. The whole job now runs as one long task. Why, and what is better?

A.coalesce(1) writes with no compression; set compression to snappyB.coalesce has no shuffle, so upstream work joins its one task; use repartition(1)C.coalesce(1) disables AQE for the job; turn AQE back onD.coalesce(1) re-reads the source; cache the source first

Discussion 0

Join free to post in the discussion.

coalesce(1) made the whole job slow

Discussion 0

Related questions