medium10 pts · first answer countsgpu-computingfsdp

How FSDP runs a layer

In PyTorch FSDP (full sharding), each GPU holds only a shard of the weights. How does it run the forward pass of a wrapped layer?

A.Sends the activations to the GPU that owns the layerB.Computes with only its own shard of the weightsC.Loads the layer from disk for every batchD.All-gathers the layer's full weights, computes, then frees them

Discussion 0

Join free to post in the discussion.

How FSDP runs a layer

Discussion 0

Related questions