hard20 pts · first answer countsdistributed-trainingllm-serving

Where to use tensor parallelism

A model is too large for one GPU. You have several 8-GPU nodes. Why do teams usually keep tensor parallelism inside a node and use pipeline or data parallelism across nodes?

A.Tensor parallelism only works with 8 GPUsB.Pipeline parallelism needs no communication at allC.Data parallelism cannot run across nodesD.Tensor parallelism exchanges activations in every layer, so it needs the fast GPU-to-GPU links inside a node

Discussion 0

Join free to post in the discussion.

Where to use tensor parallelism

Discussion 0

Related questions