hard20 pts · first answer countstransformerstraining

Where to place tensor parallelism

You train across 16 nodes of 8 GPUs each. Why do teams usually keep tensor parallelism inside a node and use pipeline or data parallelism across nodes?

A.Tensor parallelism only works when all GPUs share the same CPUB.Pipeline parallelism cannot run inside a single nodeC.It needs collectives every layer, which suits fast in-node linksD.Data parallelism needs no communication at all between nodes

Discussion 0

Join free to post in the discussion.

Where to place tensor parallelism

Discussion 0

Related questions