AI Engineer (LLM & Generative AI)
Transformers and LLM internals questions
179 questions on Transformers and LLM internals for ai engineer (llm & generative ai)s, easiest first.
Easy 55
- What is a tokeneasy
- Softmax of equal scoreseasy
- What softmax outputs look likeeasy
- How rare words get tokenizedeasy
- What LLM pretraining learnseasy
- How BPE builds its vocabularyeasy
- Tokens per English wordeasy
- Why Hindi text uses more tokenseasy
- What the end-of-sequence token doeseasy
- What the embedding layer doeseasy
- Query, key and value in attentioneasy
- Why transformers need position informationeasy
- What a decoder-only model predictseasy
- Where BERT-style models fiteasy
- What fills the context windoweasy
- Why transformers use residual connectionseasy
- Why use several attention headseasy
- Why a padding mask is neededeasy
- Why byte-level BPE never needs UNKeasy
- What the final layer outputseasy
- What the feed-forward layer seeseasy
- Why transformers normalise activationseasy
- The loss used in LLM pretrainingeasy
- Reading a perplexity numbereasy
- How training sees the previous tokenseasy
- Base model versus instruct modeleasy
- What SFT data looks likeeasy
- Role of the reward model in RLHFeasy
- What LoRA actually trainseasy
- How a mixture-of-experts layer workseasy
- Why training uses bf16easy
- What 7B in a model name meanseasy
- Memory for 7B weights in fp16easy
- What knowledge distillation doeseasy
- Fine-tuning for 20 epochs on 200 exampleseasy
- What greedy decoding doeseasy
- Sampling from the top 40 tokenseasy
- What top-p sampling doeseasy
- Reply that ends mid-sentenceeasy
- Using a stop sequenceeasy
- Why chat replies arrive word by wordeasy
- What time to first token measureseasy
- Why models are quantisedeasy
- What a repetition penalty is foreasy
- Why LLMs state false factseasy
- Learning from examples in the prompteasy
- Why output is slower than inputeasy
- Making sampled outputs repeatableeasy
- What logprobs tell youeasy
- What beam search keepseasy
- How a vision transformer reads an imageeasy
- What CLIP learnseasy
- What a knowledge cutoff meanseasy
- Where the system prompt liveseasy
- Why transformers run on GPUseasy
Medium 71
- Temperature changes probabilitiesmedium
- Parameters in the embedding matrixmedium
- Tying input and output embeddingsmedium
- Working out the head dimensionmedium
- Size of one attention score matrixmedium
- Doubling context lengthmedium
- Property of sinusoidal positionsmedium
- Limit of learned position embeddingsmedium
- How ALiBi encodes positionmedium
- Where cross-attention gets its inputsmedium
- KV cache saving from GQAmedium
- Multi-query versus grouped-query attentionmedium
- Computing an attention outputmedium
- Model paired with the wrong tokenizermedium
- Chat model given raw textmedium
- Why LLMs stumble on arithmeticmedium
- The space before a wordmedium
- What ## means in WordPiecemedium
- How SentencePiece handles spacesmedium
- Choosing a vocabulary sizemedium
- Causal mask filled with zerosmedium
- Padding side for batched generationmedium
- Monthly input-token bill in rupeesmedium
- Parameters in one transformer blockmedium
- Why the FFN hidden size is 11,008medium
- RMSNorm instead of LayerNormmedium
- Pre-norm versus post-norm blocksmedium
- Turning a loss into perplexitymedium
- Loss of an untrained modelmedium
- Compute-optimal training tokensmedium
- Estimating training FLOPsmedium
- Memory for full fine-tuning with Adammedium
- LoRA parameters for one 4,096 matrixmedium
- What QLoRA quantisesmedium
- Which tokens get loss in SFTmedium
- How DPO differs from PPO-based RLHFmedium
- Why RLHF adds a KL penaltymedium
- Active parameters in an MoE modelmedium
- Why MoE training needs load balancingmedium
- What gradient checkpointing tradesmedium
- Why training starts with warmupmedium
- Packing short examples into one sequencemedium
- What tensor parallelism splitsmedium
- Counting tokens in the nucleusmedium
- Probabilities after top-kmedium
- Which tokens min-p keepsmedium
- Beam search for open-ended chatmedium
- Presence versus frequency penaltymedium
- What limits prefill and decode speedmedium
- Upper bound on decode speedmedium
- What continuous batching improvesmedium
- What PagedAttention solvesmedium
- Reusing a long shared system promptmedium
- Memory for a 70B model at 4-bitmedium
- How speculative decoding speeds things upmedium
- Guaranteeing valid JSON outputmedium
- Banning a token with logit biasmedium
- Different answers at temperature 0medium
- Sampling several reasoning pathsmedium
- Bigger batches on an LLM servermedium
- Why step-by-step reasoning helpsmedium
- Why 4-bit weights speed up decodingmedium
- What the KV cache savesmedium
- Patch tokens in a 224 × 224 imagemedium
- Doubling image resolutionmedium
- How Whisper-style speech models workmedium
- Positives and negatives in CLIP trainingmedium
- Pooling a decoder model into one vectormedium
- Reading predictions from middle layersmedium
- One neuron, many unrelated conceptsmedium
- What sparse autoencoders findmedium
Hard 53
- Why long contexts eat GPU memoryhard
- Sizing the KV cachehard
- Why attention divides by sqrt(d_k)hard
- What the causal mask preventshard
- Counting letters in a wordhard
- Extending RoPE by position interpolationhard
- Why YaRN scales frequencies unevenlyhard
- Decoder with no positional encodinghard
- Sliding cache that drops the first tokenshard
- Reach of sliding-window attentionhard
- Subtracting the max in softmaxhard
- NaNs from a fully masked rowhard
- Tokens that make the model behave oddlyhard
- Prompt that ends with a spacehard
- Changing the number of headshard
- Left padding with absolute positionshard
- Two BOS tokens at the starthard
- Broken characters while streaminghard
- What FSDP and ZeRO-3 shardhard
- Loss spike in a long training runhard
- RLHF model becomes wordyhard
- Raising beta in DPOhard
- Doubling LoRA rank with alpha fixedhard
- Merge or keep LoRA adapters separatehard
- Teaching new facts by fine-tuninghard
- Gradients vanish in fp16 traininghard
- Running out of fresh training datahard
- Where to place tensor parallelismhard
- Reducing the pipeline bubblehard
- Token probabilities after RLHFhard
- Fine-tuned model never stops talkinghard
- Training past compute-optimalhard
- Sizing GPUs for an MoE modelhard
- Tokens gained per speculative stephard
- Greedy decoding stuck in a loophard
- High temperature combined with top-phard
- Outliers in int8 activation quantisationhard
- Switching the KV cache to FP8hard
- Field order in structured outputhard
- Long prompts slowing other usershard
- JSON parse errors on long outputshard
- Reasoning model for simple extractionhard
- Beam search prefers short outputshard
- What induction heads dohard
- Using attention maps as explanationshard
- Ordering prompts for cache hitshard
- GPU at low utilisation during decodinghard
- Repetition penalty and generated codehard
- Passing the needle test is not enoughhard
- Steering a model through its activationshard
- State-space models versus attentionhard
- Why prompt injection is hard to fixhard
- Adding domain tokens to a tokenizerhard