hard20 pts · first answer countsinferencedeployment

Serving a mixture-of-experts model

A mixture-of-experts model has many experts per layer, but each token is routed to only 2 of them. What does this mean for serving?

A.You only need GPU memory for the 2 active expertsB.Compute and memory both scale with the total parameter countC.Compute follows the active experts, but memory must hold all expertsD.The model cannot be batched because tokens use different experts

Discussion 0

Join free to post in the discussion.

Serving a mixture-of-experts model

Discussion 0

Related questions