hard20 pts · first answer countstransformersattentioncontext-window
Reach of sliding-window attention
A 32-layer model uses sliding-window attention with a window of 4,096 tokens in every layer. In theory, how far back can information travel to reach the last token?
Discussion 0
Join free to post in the discussion.
Related questions
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- Does the reply use context spaceAI Engineer (LLM & Generative AI)
- Checking prompt size in advanceAI Engineer (LLM & Generative AI)
- When you may not need retrievalAI Engineer (LLM & Generative AI)
- How BPE builds its vocabularyAI Engineer (LLM & Generative AI)
- Tokens per English wordAI Engineer (LLM & Generative AI)
- Why Hindi text uses more tokensAI Engineer (LLM & Generative AI)
- What the end-of-sequence token doesAI Engineer (LLM & Generative AI)