hard20 pts · first answer countssparkpythonperformance

A slow Python UDF in PySpark

A PySpark job applies a row-by-row Python UDF to clean phone numbers and is 10 times slower than the rest of the pipeline. What is the best first fix?

A.Rewrite it with built-in Spark SQL functions (such as regexp_replace), or a vectorised pandas UDF if logic is complexB.Call collect() and clean the data on the driverC.Add cache() after the UDFD.Increase the number of retries

← Adding partitions to a keyed topic Distinct users on billions of rows →