hard20 pts · first answer countssparkspark-sqlwindow-functions

A window with no partitionBy

A job adds a global rank with F.row_number().over(Window.orderBy("amount")) on 2 billion rows. The log warns "No Partition Defined for Window operation" and one task runs for hours. Why?

A.All rows are moved into a single partition to be orderedB.row_number() is not supported on numeric columnsC.Spark recomputes the window once per executorD.The window is evaluated on the driver instead

Discussion 0

Join free to post in the discussion.

A window with no partitionBy

Discussion 0

Related questions