hard20 pts · first answer countscost-optimisationmodel-serving

Hundreds of low-traffic models

You serve 300 small per-customer models, each with low and spiky traffic, and each has its own dedicated GPU pod. Costs are huge and GPUs sit mostly idle. What is the best direction?

A.Use multi-model serving that loads models on demand onto shared GPUs (or CPUs) with caching, plus scale-to-zero for rarely used modelsB.Give each model two GPUs for safetyC.Merge all customer data and retrain one giant model without checking qualityD.Turn off monitoring to save money

Discussion 0

Join free to post in the discussion.

Hundreds of low-traffic models

Discussion 0

Related questions