hard20 pts · first answer countstransformerstraining

Loss spike in a long training run

Three weeks into pretraining, the loss suddenly jumps and does not recover. What is the common practical response?

A.Keep training; loss spikes always recover on their own given timeB.Roll back to a checkpoint before the spike, skip the bad data, maybe lower the LRC.Restart the whole run from scratch with a new random seedD.Double the learning rate to push the model out of the bad region

Discussion 0

Join free to post in the discussion.

Loss spike in a long training run

Discussion 0

Related questions