hard20 pts · first answer countsdistributed-traininggpu-memory

Model does not fit in GPU memory

With plain data parallel training (DDP), your model plus Adam optimizer states runs out of memory on each GPU, even with batch size 1. Which approach directly addresses this?

A.Increase the number of GPUs while keeping plain DDPB.Use fully sharded data parallel (FSDP / ZeRO) to shard parameters, gradients and optimizer states across GPUsC.Increase the batch size so the GPU is used more efficientlyD.Switch from Adam to Adam with a larger learning rate

Discussion 0

Join free to post in the discussion.

Model does not fit in GPU memory

Discussion 0

Related questions