hard20 pts · first answer countsgpu-computingfsdpgpu-memory

Choosing the FSDP wrapping unit

Wrapping the entire model as a single FSDP unit gives little memory saving. Why is wrapping each transformer block separately better?

A.It removes the need for an optimizerB.Only one block's full weights are gathered at a timeC.It turns off communication between GPUsD.It makes every block run on its own node

Discussion 0

Join free to post in the discussion.

Choosing the FSDP wrapping unit

Discussion 0

Related questions