medium10 pts · first answer countssparkdelta-lake
Thousands of tiny Delta files
A Delta table written by a streaming job has 80,000 files of about 200 KB each. Queries usually filter on customer_id. What is the standard fix?
Discussion 0
Join free to post in the discussion.
Related questions
- When Spark actually runsData Engineer
- What a Spark partition isData Engineer
- What the _delta_log folder isData Engineer
- What the Spark driver doesData Engineer
- Calling collect() on 500 GBData Engineer
- Why DataFrames beat RDDs of objectsData Engineer
- Entry point of a Spark 3+ appData Engineer
- Tasks in a 64-partition stageData Engineer