medium10 pts · first answer countsoptimizationtraining
Why use learning rate warmup
Why do transformer training recipes usually ramp the learning rate up over the first few hundred or thousand steps?
Discussion 0
Join free to post in the discussion.
Related questions
- When early stopping triggersMachine Learning Engineer
- Optimizer steps in one epochMachine Learning Engineer
- Training loop without zero_gradMachine Learning Engineer
- Loss turns NaN in an RNNMachine Learning Engineer
- A learning rate that is too highAI Engineer (LLM & Generative AI)
- Why bf16 is easier than fp16MLOps Engineer