hard20 pts · first answer countstransformersinferencedecoding
Tokens gained per speculative step
A draft model proposes k = 4 tokens per step, and each is accepted independently with probability α = 0.8 (stopping at the first rejection). Expected tokens per target pass is (1 − α^(k+1)) / (1 − α). What is it?
Discussion 0
Join free to post in the discussion.
Related questions
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- How BPE builds its vocabularyAI Engineer (LLM & Generative AI)
- Tokens per English wordAI Engineer (LLM & Generative AI)
- Why Hindi text uses more tokensAI Engineer (LLM & Generative AI)
- What the end-of-sequence token doesAI Engineer (LLM & Generative AI)
- What the embedding layer doesAI Engineer (LLM & Generative AI)
- Query, key and value in attentionAI Engineer (LLM & Generative AI)
- Why transformers need position informationAI Engineer (LLM & Generative AI)