AI Engineer (LLM & Generative AI)
Model serving and inference questions
44 questions on Model serving and inference for ai engineer (llm & generative ai)s, easiest first.
Easy 6
Medium 19
- Memory for 4-bit weightsmedium
- Memory after int8 quantisationmedium
- Binary quantisation with rescoringmedium
- How product quantisation saves memorymedium
- KV cache saving from GQAmedium
- Multi-query versus grouped-query attentionmedium
- Padding side for batched generationmedium
- Active parameters in an MoE modelmedium
- What limits prefill and decode speedmedium
- Upper bound on decode speedmedium
- What continuous batching improvesmedium
- What PagedAttention solvesmedium
- Reusing a long shared system promptmedium
- Memory for a 70B model at 4-bitmedium
- How speculative decoding speeds things upmedium
- Different answers at temperature 0medium
- Bigger batches on an LLM servermedium
- Why 4-bit weights speed up decodingmedium
- What the KV cache savesmedium
Hard 19
- Why long contexts eat GPU memoryhard
- What streaming improveshard
- Sizing the KV cachehard
- How speculative decoding helpshard
- Continuous batching in servinghard
- Serving a mixture-of-experts modelhard
- How many candidates to rescorehard
- Sliding cache that drops the first tokenshard
- Left padding with absolute positionshard
- Broken characters while streaminghard
- Merge or keep LoRA adapters separatehard
- Sizing GPUs for an MoE modelhard
- Tokens gained per speculative stephard
- Outliers in int8 activation quantisationhard
- Switching the KV cache to FP8hard
- Long prompts slowing other usershard
- Reasoning model for simple extractionhard
- Ordering prompts for cache hitshard
- GPU at low utilisation during decodinghard