Distributed Training
9 practice items · needed by 2 roles
Easy 0Medium 3Hard 6
Practice questions
- Effective batch size across GPUsmedium
- Model does not fit in GPU memoryhard
- What ZeRO stage 3 shardshard
- Activation checkpointing trade-offhard
- Training on spot GPUsmedium
- Speeding up training on 8 GPUsmedium
- Where to use tensor parallelismhard
- Memory to fully fine-tune 7B with Adamhard
- Loss spike after resuming traininghard