hard20 pts · first answer countsml-pipelinesreproducibilitydistributed-training

Different GPU count, different result

A distributed training run on 8 GPUs is reproduced on 4 GPUs with the same seed and data. Results differ noticeably. Why?

A.Shards, batches and reduction order depend on GPU countB.Seeds only work when there are exactly 8 GPUs presentC.Fewer GPUs always produce a higher final accuracyD.PyTorch ignores the seed entirely in distributed mode

Discussion 0

Join free to post in the discussion.

Different GPU count, different result

Discussion 0

Related questions