medium10 pts · first answer countstransformerstraining

What tensor parallelism splits

In large-model training, what does tensor parallelism split across GPUs?

A.The weight matrices inside each layer, so GPUs compute parts of each matmulB.The batch, so each GPU holds a full model copy and different dataC.The layers, so each GPU holds a consecutive group of layersD.The vocabulary, so each GPU trains a different tokenizer

Discussion 0

Join free to post in the discussion.

What tensor parallelism splits

Discussion 0

Related questions