medium10 pts · first answer countstransformerstokenization

Model paired with the wrong tokenizer

A developer loads model weights from one checkpoint but the tokenizer from a different model family. Generation runs without errors but the output is nonsense. Why?

A.Token ids now map to different strings than the ones the model learnedB.The tokenizer is only used for training, so it should not matterC.Different tokenizers produce different context window sizesD.The tokenizer controls temperature, which is now set too high

Discussion 0

Join free to post in the discussion.

Model paired with the wrong tokenizer

Discussion 0

Related questions