hard20 pts · first answer countsgpu-computingmixed-precision

Pure BF16 training stalls

A team stores weights and optimizer states only in BF16 to save memory. Loss stops improving even though gradients look fine. What is the likely cause?

A.BF16 overflows above 65,504B.Small updates round away in BF16 weightsC.BF16 cannot be used with the Adam optimizerD.The GPU does not support BF16 matmuls

Discussion 0

Join free to post in the discussion.

Pure BF16 training stalls

Discussion 0

Related questions