hard20 pts · first answer countstransformerstrainingscaling-laws

Running out of fresh training data

You have 500B tokens of good data but your compute budget suits 2T tokens. What does research on data-constrained scaling suggest?

A.Up to about 4 epochs of repeats is nearly as good as fresh data; far more helps littleB.Any repeated token makes the model worse than training on 500B onceC.Repeating data 100 times works as well as 100 times more fresh dataD.Repeats only help if every copy is shuffled into a new language

Discussion 0

Join free to post in the discussion.

Running out of fresh training data

Discussion 0

Related questions