medium10 pts · first answer countsdistributed-training

Speeding up training on 8 GPUs

The model fits easily on one GPU, but training takes too long. You have 8 GPUs. Which approach fits best?

A.Data parallelism: each GPU holds a copy and processes a different slice of each batchB.Pipeline parallelism across the 8 GPUsC.Tensor parallelism that splits every layerD.Train on one GPU and leave the rest idle

Discussion 0

Join free to post in the discussion.

Speeding up training on 8 GPUs

Discussion 0

Related questions