hard20 pts · first answer countstransformerstokenization
Two BOS tokens at the start
Your pipeline renders the chat template (which already starts with <|begin_of_text|>) and then tokenizes it with add_special_tokens=True. Quality drops slightly. Why?
Discussion 0
Join free to post in the discussion.
Related questions
- What is a tokenAI Engineer (LLM & Generative AI)
- How rare words get tokenizedAI Engineer (LLM & Generative AI)
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- How BPE builds its vocabularyAI Engineer (LLM & Generative AI)
- Tokens per English wordAI Engineer (LLM & Generative AI)
- Why Hindi text uses more tokensAI Engineer (LLM & Generative AI)
- What the end-of-sequence token doesAI Engineer (LLM & Generative AI)
- What the embedding layer doesAI Engineer (LLM & Generative AI)