medium10 pts · first answer countssparkspark-sql

Order of values from collect_list

A job sorts by event_time, then does groupBy("user_id").agg(F.collect_list("page")). The page lists are sometimes out of time order. Why?

A.collect_list always returns values in reverseB.The groupBy shuffle does not keep the earlier sort orderC.collect_list sorts values alphabeticallyD.The sort is applied only on the driver

Discussion 0

Join free to post in the discussion.

Order of values from collect_list

Discussion 0

Related questions