easy5 pts · first answer countstransformerstraining

How training sees the previous tokens

During pretraining, when the model predicts token 50, what does it use as tokens 1 to 49?

A.The real tokens from the text, all positions at onceB.The tokens it generated itself one step at a timeC.Random tokens, so it learns to be robust to noiseD.Only token 49, because earlier tokens are thrown away

Discussion 0

Join free to post in the discussion.

How training sees the previous tokens

Discussion 0

Related questions