hard20 pts · first answer countssparkspark-sql

first() gives different answers

groupBy("customer").agg(F.first("plan")) returns different plans for the same customer on different runs of the same job. Why?

A.first() picks a random row by design on every single runB.The job reads a sample of the data each timeC.first() returns the last row when AQE is onD.Row order after a shuffle is not fixed, so first() varies

Discussion 0

Join free to post in the discussion.

first() gives different answers

Discussion 0

Related questions