hard20 pts · first answer countsmixed-precisiondeep-learning

Why fp16 needs loss scaling but bf16 usually does not

Mixed precision training with fp16 often needs dynamic loss scaling, while bf16 training usually works without it. Why?

A.bf16 has more mantissa bits than fp32B.fp16 does not support matrix multiplication on GPUsC.bf16 has the same exponent range as fp32, so small gradients do not underflow to zeroD.bf16 automatically averages gradients across devices

Discussion 0

Join free to post in the discussion.

Why fp16 needs loss scaling but bf16 usually does not

Discussion 0

Related questions