Transformers and Attention
Transformers use self-attention to weigh input tokens dynamically. They have become the foundation of large language models and modern sequence modeling.
Key Points
- Self-attention computes $\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$.
- Multi-head attention allows the model to attend to different representation subspaces.
- Positional encodings inject sequence order information.
Formulas
Scaled dot-product attention
$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$
Multi-head attention
$$\operatorname{MultiHead}(Q,K,V) = \operatorname{Concat}(\operatorname{head}_1, \dots, \operatorname{head}_h) W^O$$
Positional encoding
$$PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d})$$
Code Example
import torch
import torch.nn.functional as F
Q = torch.randn(8, 64) # seq_len, d_k
K = torch.randn(8, 64)
V = torch.randn(8, 64)
scores = Q @ K.T / (64 ** 0.5)
attn_weights = F.softmax(scores, dim=-1)
out = attn_weights @ V