easy5 pts · first answer countssparkspark-driver
Calling collect() on 500 GB
A developer runs df.collect() on a DataFrame with about 500 GB of data to "look at it". What is the most likely result?
rows = df.collect() print(rows[:10])
Discussion 0
Join free to post in the discussion.
Related questions
- When Spark actually runsData Engineer
- What a Spark partition isData Engineer
- What the Spark driver doesData Engineer
- Why DataFrames beat RDDs of objectsData Engineer
- Entry point of a Spark 3+ appData Engineer
- Tasks in a 64-partition stageData Engineer
- Where a job splits into stagesData Engineer
- Cost of inferSchema on CSVData Engineer