medium10 pts · first answer countssparkpyspark
Union of files with a new column
Last month's DataFrame has columns (id, amount). This month's has (id, amount, gst). You want one DataFrame with all three columns and null gst for old rows. What do you use?
Discussion 0
Join free to post in the discussion.
Related questions
- When Spark actually runsData Engineer
- What a Spark partition isData Engineer
- What the Spark driver doesData Engineer
- Calling collect() on 500 GBData Engineer
- Why DataFrames beat RDDs of objectsData Engineer
- Entry point of a Spark 3+ appData Engineer
- Tasks in a 64-partition stageData Engineer
- Where a job splits into stagesData Engineer