hard20 pts · first answer countsmodel-servingcostreliability
Spot GPUs for an online service
To cut costs, you consider running an online LLM API on spot GPUs that can be reclaimed with about two minutes' notice. What design makes this safe?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- GPU bill keeps risingMLOps Engineer
- Where to write checkpointsMLOps Engineer
- Cost of an 8-GPU node for a dayMLOps Engineer
- Matching the GPU to the jobMLOps Engineer