easy5 pts · first answer countstransformerstokenization

Why Hindi text uses more tokens

The same customer message written in Hindi (Devanagari) uses about three times as many tokens as its English version. What is the most likely reason?

A.The tokenizer was trained mostly on English, so Hindi words split into more piecesB.Hindi sentences always contain three times as many words as English onesC.The model translates Hindi to English internally before tokenizing itD.Devanagari letters are each stored as three separate special tokens

Discussion 0

Join free to post in the discussion.

Why Hindi text uses more tokens

Discussion 0

Related questions