hard20 pts · first answer countsattentiontransformers

Why attention divides by sqrt(d_k)

Scaled dot-product attention computes softmax(QK^T / sqrt(d_k)). What goes wrong if you drop the sqrt(d_k) scaling for large head dimensions?

A.The attention matrix stops being square, so the shapes no longer matchB.The model can no longer apply a causal mask during trainingC.Dot products grow with d_k, softmax saturates toward one-hot, and gradients become tinyD.Attention becomes linear in sequence length instead of quadratic

Discussion 0

Join free to post in the discussion.

Why attention divides by sqrt(d_k)

Discussion 0

Related questions