medium10 pts · first answer countsdeep-learningoptimisation
Early layers stop learning
In a very deep network using sigmoid activations, the first layers barely change during training. Which change most directly helps?
Discussion 0
Join free to post in the discussion.
Related questions
- Symptom of a learning rate that is too highMachine Learning Engineer
- Different results every inference runMachine Learning Engineer
- Initialisation for a deep ReLU networkMachine Learning Engineer
- Why fp16 needs loss scaling but bf16 usually does notMachine Learning Engineer
- L2 penalty versus AdamWMachine Learning Engineer
- Why activation functions existAI Engineer (LLM & Generative AI)
- A learning rate that is too highAI Engineer (LLM & Generative AI)