MLOps Engineer
Model serving and inference questions
112 questions on Model serving and inference for mlops engineers, easiest first.
Easy 33
- When batch inference fitseasy
- Traffic before the model has loadedeasy
- Memory for 7B parameters in fp16easy
- Why quantise a modeleasy
- Why export a model to ONNXeasy
- Protocol for internal inference callseasy
- Latency and throughputeasy
- What TTFT measureseasy
- Gap between streamed tokenseasy
- Two phases of LLM generationeasy
- Requests that take five minuteseasy
- Knowing which model is liveeasy
- Rolling out a new model safelyeasy
- Testing on live traffic invisiblyeasy
- Requests hang forevereasy
- What vLLM is used foreasy
- Purpose of Triton Inference Servereasy
- Limiting output lengtheasy
- Setting temperature to 0easy
- Same question asked againeasy
- Hardware for a small tree modeleasy
- Loading weights from an unknown sourceeasy
- Same code, different outputs after rebuildeasy
- Why blue-green suits model releaseseasy
- One client floods the endpointeasy
- Logging prompts in productioneasy
- Dashboard for an LLM serviceeasy
- Running a model on a laptopeasy
- Huge inputs crash the servereasy
- Serverless GPU for low trafficeasy
- Model inside the web app processeasy
- When a GPU is worth iteasy
- Health check passes but model failseasy
Medium 43
- Low GPU use under loadmedium
- KV cache size for one long requestmedium
- Serving many fine-tuned variantsmedium
- Smaller model with similar qualitymedium
- Splitting requests into batchesmedium
- Silent wrong predictions from bad inputmedium
- Reusing a shared system promptmedium
- Long prompts stall other usersmedium
- Concurrent requests from Little's lawmedium
- Total output tokens per secondmedium
- vLLM gpu_memory_utilizationmedium
- What AWQ and GPTQ quantisemedium
- FP8 for servingmedium
- Quantising the KV cachemedium
- Prefix cache hit rate is lowmedium
- Guaranteeing valid JSONmedium
- Model keeps writing extra turnsmedium
- What top_p=0.9 meansmedium
- Beam search in a chat APImedium
- Setting a latency SLOmedium
- KV cache fills during a burstmedium
- GPU idle while CPU is maxedmedium
- TensorRT engine fails on new GPUsmedium
- Streaming arrives all at oncemedium
- Retry storm after an outagemedium
- Primary model keeps timing outmedium
- Embedding one document at a timemedium
- Cross-encoder reranker is slowmedium
- First requests after deploy are slowmedium
- Variable input lengths in TensorRTmedium
- Triton max_queue_delay settingmedium
- Small model leaves the GPU idlemedium
- Many small models on one A100medium
- Many adapters on one vLLM servermedium
- Cost per million tokens on own GPUmedium
- GPU hours for an offline jobmedium
- vLLM fails to start: KV too smallmedium
- KV cache with sliding-window layersmedium
- Accuracy slips months after launchmedium
- Same model, worse online scoresmedium
- Request rejected for context lengthmedium
- Users switch models mid-sessionmedium
- Pre-processing on the client is slowmedium
Hard 36
- p99 latency spikes during traffic burstshard
- Why LLM servers use continuous batchinghard
- Hundreds of low-traffic modelshard
- Where to use tensor parallelismhard
- KV cache with grouped-query attentionhard
- Batches of mixed text lengthshard
- Accuracy fell after int8 quantisationhard
- State leaking between requestshard
- Semantic cache returns wrong answershard
- Upper bound on single-stream speedhard
- Why batching raises decode throughputhard
- Estimating prefill timehard
- How many 8K chats fithard
- Tokens gained per verify stephard
- Speculative decoding under heavy loadhard
- Where 4-bit weights speed things uphard
- Separate prefill and decode GPUshard
- Why servers capture CUDA graphshard
- Same prompt, temperature 0, different texthard
- Autoscaling an LLM deploymenthard
- New replicas take four minuteshard
- Splitting a model across two nodeshard
- Slow first token for Dubai usershard
- Savings from cached prompt tokenshard
- When decode stops being memory-boundhard
- Throughput versus goodputhard
- How SmoothQuant enables INT8hard
- Speculation without a separate draft modelhard
- Why a 100K prompt is so slowhard
- Rolling update stuck in Pendinghard
- Spot GPUs for an online servicehard
- Errors only on very long documentshard
- Sizing for RAG versus chathard
- Placing experts on many GPUshard
- Latency explodes near full loadhard
- Overload during a sale eventhard