hard20 pts · first answer countsml-pipelinesdata-versioning

Hashing 50 TB every run

Each run computes a SHA-256 over 50 TB of Parquet files to fingerprint the dataset, adding hours. What gives a reliable fingerprint much faster?

A.Hash only the first Parquet file in the folderB.Hash a manifest of paths, sizes and object versionsC.Use the folder name itself as the fingerprintD.Skip the fingerprint and log only the run date

Discussion 0

Join free to post in the discussion.

Hashing 50 TB every run

Discussion 0

Related questions