medium10 pts · first answer countsgpu-computingfsdp

Gradient clipping with FSDP

With FSDP, calling torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) on each rank gives wrong results. Why?

A.Clipping is not allowed in mixed precisionB.FSDP gradients are always zeroC.Each rank sees only its own gradient shardD.Clipping must happen before backward

Discussion 0

Join free to post in the discussion.

Gradient clipping with FSDP

Discussion 0

Related questions