easy5 pts · first answer countstokensmultilingual-prompting

Same text, more tokens

The same short message uses noticeably more tokens in Hindi or Arabic than in English. What explains this?

A.Tokenizers are often trained mostly on English, so other scripts get split into more, smaller piecesB.Non-English text is always longer in meaningC.The API charges extra tax for other languagesD.The model translates everything to English first and counts both

Discussion 0

Join free to post in the discussion.

Same text, more tokens

Discussion 0

Related questions