easy5 pts · first answer countsllm-fundamentalstokens

Hello and hello are not the same

In most LLM tokenizers, are "Hello", " hello" and "HELLO" the same token?

A.Yes; tokenizers convert all text to lower case firstB.Yes; spaces and capitals are removed before tokenizingC.No; case and a leading space usually give different tokensD.No; but only because each letter is always its own token

Discussion 0

Join free to post in the discussion.

Hello and hello are not the same

Discussion 0

Related questions