hard20 pts · first answer countstransformerstraining

What FSDP and ZeRO-3 shard

Plain data parallelism keeps a full copy of weights, gradients and optimiser states on every GPU. What does FSDP (or ZeRO stage 3) change?

A.It shards only the input batch, keeping full copies of everything elseB.It shards all three across GPUs and gathers weights only when a layer needs themC.It moves the whole optimiser to the CPU and keeps weights on one GPUD.It removes gradients entirely by using forward-only training

Discussion 0

Join free to post in the discussion.

What FSDP and ZeRO-3 shard

Discussion 0

Related questions