Let tokens build context from one another
Self-attention lets each token weight information from other tokens when building a context-aware representation.
The idea
For each word, ask which other words matter most right now, assign them weights, then mix their information accordingly.
Queries and keys produce scaled dot-product scores. Softmax converts each score row into attention weights, which form a weighted combination of value vectors.
How it works
- Project token representations into Q, K, and V spaces.
- Compute query-key similarity scores and scale by √d_k.
- Apply masks if needed, then softmax each score row.
- Use the normalized weights to form weighted sums of Value vectors; repeat across heads and project outputs.
Scaled dot-product attention
Scaling controls score magnitude as key dimension grows; softmax creates normalized positive weights; those weights mix Value vectors.
Attention(Q,K,V) = softmax(QKᵀ / √d_k) V
Check your understanding
What does softmax do to one row of attention scores?
- Turns it into nonnegative weights that sum to 1
- Deletes the values
- Builds a decision tree
- Chooses the train/test split
Answer and explanation
Softmax converts a row of scores into normalized positive weights whose sum is 1, which are then used to mix value vectors.
A common misconception
Self-attention is not the whole Transformer. A Transformer block also includes projections, residual pathways, normalization, feed-forward networks, positional information, and other architecture details.
Deep Learning
Build intuition for weighted neurons, shared filters, recurrent state, and self-attention before opening the full labs.