hard20 pts · first answer countstransformersattention

NaNs from a fully masked row

Training is fine until a batch contains an example that is entirely padding. The loss becomes NaN. Your mask fills masked scores with −inf. What is the likely cause?

A.−inf is converted to +inf in bf16, so pad tokens get all the weightB.Padding tokens have no embeddings, so the lookup returns NaNC.The learning rate is too high only for padded examplesD.A row where every score is −inf gives softmax 0/0, which is NaN

Discussion 0

Join free to post in the discussion.

NaNs from a fully masked row

Discussion 0

Related questions