hard20 pts · first answer countsmodel-servingspeculative-decodingbatching

Speculative decoding under heavy load

Speculative decoding gives a 2× speed-up at low traffic, but at peak load with large batches the gain nearly disappears. Why?

A.Draft models stop working when more users connectB.Large batches disable the KV cacheC.At large batch, compute is busy, so wasted checks cost real workD.The target model changes its weights at peak times

Discussion 0

Join free to post in the discussion.

Speculative decoding under heavy load

Discussion 0

Related questions