easy5 pts · first answer countstokenization

How rare words get tokenized

A subword tokenizer (such as BPE) meets a rare word like "unbelievability". What usually happens?

A.It is split into several known subword piecesB.It is replaced by an unknown-word token and lostC.It becomes one new token added to the vocabularyD.The request fails with a tokenization error

Discussion 0

Join free to post in the discussion.

How rare words get tokenized

Discussion 0

Related questions