hard20 pts · first answer countsdistributed-training
What ZeRO stage 3 shards
With DeepSpeed ZeRO stage 3 or PyTorch FSDP full sharding, what is split across data-parallel GPUs?
Discussion 0
Join free to post in the discussion.
Related questions
- Effective batch size across GPUsMachine Learning Engineer
- Model does not fit in GPU memoryMachine Learning Engineer
- Activation checkpointing trade-offMachine Learning Engineer
- Training on spot GPUsMLOps Engineer
- Speeding up training on 8 GPUsMLOps Engineer
- Where to use tensor parallelismMLOps Engineer
- Memory to fully fine-tune 7B with AdamMLOps Engineer
- Loss spike after resuming trainingMLOps Engineer