easy5 pts · first answer countsgpu-computingcollectives

Purpose of all-reduce in DDP

In DDP, what does the all-reduce step after backward do?

A.Sends all the weights to GPU 0 onlyB.Splits the batch into equal partsC.Gives every GPU the averaged gradientsD.Saves a checkpoint on every GPU

Discussion 0

Join free to post in the discussion.