Transformer Self-Attention
Trace Q, K, V and token-to-token attention without pretending to train an LLM.
This focused lab teaches the self-attention mechanism with small deterministic vectors so every matrix remains inspectable.
Step by step
- Represent tokens as vectors.
- Project into Q, K and V.
- Calculate scaled query-key scores.
- Apply stable softmax.
- Use attention weights to mix value vectors.
Core formulas
Scaled attention
A = softmax(QKᵀ / √dₖ)Each query gets a normalized distribution over keys.
Output
O = A·VValues are mixed with the calculated weights.
When to use Transformer Self-Attention
- Understanding the core self-attention operation.