hard20 pts · first answer countsdata-lakeperformance

Millions of tiny files

A streaming job writes to a data lake table every 30 seconds. After a few months, queries are very slow even though total data is only 500 GB. What is the most likely cause and fix?

A.Too many small files cause high metadata and open-file overhead; run compaction (for example OPTIMIZE in Delta Lake) and larger trigger intervalsB.Parquet is too slow; switch to CSVC.The data needs more partitions by secondD.The cluster has too much memory

← Exactly-once end to end Safe backfills →