hard20 pts · first answer countstransformerstokenizationinference

Broken characters while streaming

Your app streams tokens to the browser and decodes each token on its own. Arabic text and emoji sometimes show as "�" mid-stream. What is the cause and fix?

A.The model outputs Latin-1 instead of UTF-8 for non-English textB.The browser cannot show Arabic, so it needs a special fontC.One character can span several tokens; buffer bytes until it is completeD.The tokenizer drops emoji, so the model must be fine-tuned on them

Discussion 0

Join free to post in the discussion.

Broken characters while streaming

Discussion 0

Related questions