Explain in simple terms how self-attention works in a transformer.

What the interviewer wants to hear

• Each token creates query, key and value vectors
• Scores are query-key dot products scaled by sqrt(d_k)
• Softmax turns scores into weights over all tokens
• Output is the weighted sum of value vectors
• Multiple heads learn different relationships; a causal mask hides future tokens

Model answer

Self-attention lets every token look at every other token and decide which ones matter for it. Each token embedding is projected into three vectors: a query, a key and a value. To update one token, we take the dot product of its query with every key, divide by the square root of the key size to keep values stable, and apply softmax to get attention weights that sum to one. The new representation is the weighted sum of the value vectors. So in "the animal did not cross the street because it was tired", the token "it" can put high weight on "animal". Multi-head attention runs several of these in parallel so different heads can capture different relations. In decoder models a causal mask stops tokens from attending to future positions.

More easy Machine Learning Engineer questions

• What is overfitting, how do you detect it, and how do you reduce it?
• Explain precision and recall with a real example. When would you favour one over the other?
• How does gradient descent work, and what is the role of the learning rate?
• What are the training, validation and test sets used for, and how do you split them?
• Why should you build a simple baseline before a complex model? Give examples of good baselines.
• Explain the bias-variance trade-off and how it guides your choice of model.
• What is an embedding, and how would you use embeddings in a product feature?
• When would you choose a tree-based model like gradient boosting over a linear model, and when not?
• How do you handle missing values in a dataset before training a model?