Practiced one out loud? Paste your answer into the free self-check and get a score in 30 seconds
What the interviewer is testing
Whether you can explain the query-key-value computation, why self-attention improves long-range token interactions, the quadratic cost, and the reason for scaling before softmax.
Answer framework
- Represent each token with three learned projections: a query, a key and a value.
- Compute compatibility scores with the query-key dot product, scale them by 1/sqrt(d_k), and apply softmax to obtain attention weights.
- Produce each token's output as a weighted sum of the value vectors using those attention weights.
- Allow each token to retrieve information from any position permitted by the attention mask in one layer, avoiding a recurrent bottleneck while supporting parallel sequence processing during training.
- Recognize the trade-off: full self-attention has O(n^2) compute and an attention matrix that also scales quadratically with sequence length.
Sample answer
I explain self-attention as a way for the model to decide which other tokens in the sequence matter for the token it is processing. Each token is projected into a query, a key and a value. The query represents what the token is looking for, the key represents what it offers for matching, and the value carries information that can be passed along. We compute scores by taking the query-key dot product, divide by the square root of the key dimension, and apply softmax to get normalized attention weights.
The output for each position is a weighted sum of the value vectors using those weights. The important benefit is direct information flow: each token can pull information from positions permitted by the attention mask through one layer instead of passing it through recurrent steps. Unmasked attention can access the entire sequence; causal attention can access only current and earlier positions. During training, positions can still be processed in parallel.
The trade-off is quadratic computation. For sequence length n and fixed head dimensions, full attention uses O(n^2) compute. Explicitly storing its attention matrix also takes O(n^2) memory, but implementations such as FlashAttention avoid materializing that matrix. Dividing scores by the square root of the key dimension also matters: under standard assumptions, dot-product variance grows with dimension, which can saturate softmax and produce small gradients. Scaling helps keep scores in a useful range.
Likely follow-ups
Common mistakes
- Assuming a token can only use nearby context, as in a fixed local window.
- Forgetting to scale the QK^T scores by sqrt(d_k) before applying softmax.
- Claiming the attention cost is linear in sequence length; the full attention matrix is quadratic.
- Describing the output as a plain average of values and ignoring the learned, content-dependent weights.
The 0 to 1 AI Engineer Interview Playbook covers the answer frameworks, system-design walkthroughs and an 8-week plan.
Get the playbookSmaller start: Score your answers against 50 rubrics, $9. 50 AI engineer questions, one rubric each, PDF on Gumroad.
Interviewing for PM or TPM? See the full PM package, $59 on Gumroad: a resume system, a PM, TPM and PGM interview handbook, a 1:1 cheat sheet and a PM playbook. PM and TPM content, not AI engineer prep.
Related questions
Based on question #2 in the full question bank.