hard20 pts · first answer countssparkspark-query-plansshuffle
Shuffles after repartition on the same key
How many shuffle (Exchange) steps does the physical plan of this code contain, with AQE off and no prior partitioning?
out = (df.repartition(200, "customer_id")
.groupBy("customer_id")
.agg(F.sum("amount")))Discussion 0
Join free to post in the discussion.
Related questions
- When Spark actually runsData Engineer
- What a Spark partition isData Engineer
- What the Spark driver doesData Engineer
- Calling collect() on 500 GBData Engineer
- Why DataFrames beat RDDs of objectsData Engineer
- Entry point of a Spark 3+ appData Engineer
- Tasks in a 64-partition stageData Engineer
- Where a job splits into stagesData Engineer