hard20 pts · first answer countsmodel-servingautoscalingcold-start
New replicas take four minutes
Each new replica of a 70B model downloads 140 GB of weights from object storage at about 1 GB/s before serving. What is the most effective way to cut start-up time?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Latency and throughputMLOps Engineer
- What TTFT measuresMLOps Engineer
- Gap between streamed tokensMLOps Engineer
- Two phases of LLM generationMLOps Engineer