hard20 pts · first answer countsbatchinginference-optimisation
Batches of mixed text lengths
An NLP model batches requests dynamically. Most inputs are 50 tokens, but a few are 2,000 tokens, and throughput is poor with occasional out-of-memory errors. What helps most?
Discussion 0
Join free to post in the discussion.