medium10 pts · first answer countsdistributed-training
Effective batch size across GPUs
You train with data parallelism on 8 GPUs, a per-GPU batch of 16 and gradient accumulation of 4 steps. What is the effective batch size per optimizer step?
Discussion 0
Join free to post in the discussion.
Related questions
- Model does not fit in GPU memoryMachine Learning Engineer
- What ZeRO stage 3 shardsMachine Learning Engineer
- Activation checkpointing trade-offMachine Learning Engineer
- Training on spot GPUsMLOps Engineer
- Speeding up training on 8 GPUsMLOps Engineer
- Where to use tensor parallelismMLOps Engineer
- Memory to fully fine-tune 7B with AdamMLOps Engineer
- Loss spike after resuming trainingMLOps Engineer