hard20 pts · first answer countstransformersmixture-of-expertsinference

Sizing GPUs for an MoE model

A teammate says: "This MoE model has 47B total but only 13B active parameters, so it fits on the same GPU as a 13B dense model." What is wrong with this?

A.Nothing; only active parameters ever need to be loadedB.MoE models need more memory than their total parameter countC.All 47B must be resident, since any token may route to any expertD.Active parameters are always larger than the total in practice

Discussion 0

Join free to post in the discussion.

Sizing GPUs for an MoE model

Discussion 0

Related questions