easy5 pts · first answer countsllm-basicstokenization
What is a token
When people say an LLM has a "128k token context window", what is a token?
Discussion 0
Join free to post in the discussion.
Related questions
- Effect of temperatureAI Engineer (LLM & Generative AI)
- What the context window limitsAI Engineer (LLM & Generative AI)
- What a hallucination isAI Engineer (LLM & Generative AI)
- Temperature close to zeroAI Engineer (LLM & Generative AI)
- How rare words get tokenizedAI Engineer (LLM & Generative AI)
- What LLM pretraining learnsAI Engineer (LLM & Generative AI)
- Temperature 0 is not deterministicAI Engineer (LLM & Generative AI)
- Counting letters in a wordAI Engineer (LLM & Generative AI)