hard20 pts · first answer countslatency
Estimating response time
A model has a time to first token of 0.8 seconds and then generates 60 output tokens per second. Roughly how long does a 600-token answer take to finish?
Discussion 0
Join free to post in the discussion.
Related questions
- Making prompt caching workAI Engineer (LLM & Generative AI)
- What streaming improvesAI Engineer (LLM & Generative AI)
- Where generation time goesAI Engineer (LLM & Generative AI)
- How speculative decoding helpsAI Engineer (LLM & Generative AI)
- Percentiles vs average latencySite Reliability Engineer
- p99 latency across all instancesSite Reliability Engineer
- Combining percentiles from many serversSite Reliability Engineer
- Tail latency with fan-outSite Reliability Engineer