easy5 pts · first answer countsmodel-servingreliability
Huge inputs crash the server
Someone sends a 50 MB JSON body to your text model endpoint, and the server runs out of memory. What simple guard should be in place?
Discussion 0
Join free to post in the discussion.
Related questions
- When batch inference fitsMLOps Engineer
- Traffic before the model has loadedMLOps Engineer
- Why export a model to ONNXMLOps Engineer
- Protocol for internal inference callsMLOps Engineer
- Where to write checkpointsMLOps Engineer
- Latency and throughputMLOps Engineer
- What TTFT measuresMLOps Engineer
- Gap between streamed tokensMLOps Engineer