hard20 pts · first answer countsdistributed-trainingreproducibility

Loss spike after resuming training

A job resumed from a checkpoint that contained only model weights. Loss spiked and took hours to recover. What should checkpoints also contain?

A.Only the model weights in a smaller formatB.The Docker imageC.The validation setD.Optimiser state, learning rate scheduler state, step count, random number generator states and data loader position

Discussion 0

Join free to post in the discussion.

Loss spike after resuming training

Discussion 0

Related questions