hard20 pts · first answer countstransformersattention
Parameters in a multi-head attention block
A multi-head attention block has d_model = 512 and 8 heads, with Q, K, V and output projections and no biases. How many parameters does it have?
Discussion 0
Join free to post in the discussion.
Related questions
- Scaling factor in attentionMachine Learning Engineer
- Why RoPE encodes relative positionMachine Learning Engineer
- What FlashAttention changesMachine Learning Engineer
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- Why long contexts eat GPU memoryAI Engineer (LLM & Generative AI)
- Sizing the KV cacheAI Engineer (LLM & Generative AI)
- Why attention divides by sqrt(d_k)AI Engineer (LLM & Generative AI)
- What the causal mask preventsAI Engineer (LLM & Generative AI)