hard20 pts · first answer countstransformersattention

Subtracting the max in softmax

Production softmax kernels compute exp(x − max(x)) instead of exp(x). Why is this safe and why is it done?

A.Softmax is unchanged by subtracting a constant, and it stops exp() from overflowingB.It makes the largest probability exactly 1 so decoding is greedyC.It changes the distribution slightly but makes training fasterD.It is only needed when the vocabulary has more than 65,536 tokens

Discussion 0

Join free to post in the discussion.

Subtracting the max in softmax

Discussion 0

Related questions