medium10 pts · first answer countstransformersattention
Scaling factor in attention
In scaled dot-product attention with key dimension d_k = 64, the raw scores QK^T are divided by what?
Discussion 0
Join free to post in the discussion.
Related questions
- Parameters in a multi-head attention blockMachine Learning Engineer
- Why RoPE encodes relative positionMachine Learning Engineer
- What FlashAttention changesMachine Learning Engineer
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- Why long contexts eat GPU memoryAI Engineer (LLM & Generative AI)
- Sizing the KV cacheAI Engineer (LLM & Generative AI)
- Why attention divides by sqrt(d_k)AI Engineer (LLM & Generative AI)
- What the causal mask preventsAI Engineer (LLM & Generative AI)