medium10 pts · first answer countstransformersattentioninference

Multi-query versus grouped-query attention

Multi-query attention (MQA) uses a single key/value head shared by all query heads. Why did many newer models choose grouped-query attention (GQA) instead?

A.GQA needs no KV cache at all during generationB.MQA cannot be used with a causal mask in decoder modelsC.GQA keeps most of MQA's memory saving with less loss in qualityD.GQA lets each query head see a different context length

Discussion 0

Join free to post in the discussion.

Multi-query versus grouped-query attention

Discussion 0

Related questions