hard20 pts · first answer countstransformerstraining

Gradients vanish in fp16 training

Mixed-precision training in fp16 shows many gradients becoming exactly zero. The same run in bf16 is fine. What is the standard fp16 fix?

A.Lower the learning rate until the gradients become non-zeroB.Switch the optimiser from Adam to plain SGD with momentumC.Dynamic loss scaling: scale the loss up, unscale the gradientsD.Remove LayerNorm so that activations become larger

Discussion 0

Join free to post in the discussion.

Gradients vanish in fp16 training

Discussion 0

Related questions