Machine Learning Engineer
Transformers and LLM internals questions
31 questions on Transformers and LLM internals for machine learning engineers, easiest first.
Easy 8
Medium 15
- Scaling factor in attentionmedium
- Activation in BERT and GPT feed-forwardsmedium
- Softmax by handmedium
- Adding 100 to every logitmedium
- What LayerNorm normalises overmedium
- Doubling the context lengthmedium
- Why GPT uses a causal maskmedium
- Attention without position informationmedium
- What the KV cache storesmedium
- Where cross-attention gets keysmedium
- Active parameters in a mixture of expertsmedium
- Adding special tokensmedium
- Hindi and Arabic use more tokensmedium
- Adding tokens for an Indic scriptmedium
- LoRA cannot learn new tokensmedium