Machine Learning / DL / LLM Difficulty: Advanced

Neural Networks and Backpropagation

Neural networks are compositions of parameterized nonlinear functions. Backpropagation efficiently computes gradients using the chain rule, enabling training of deep models.

Key Points

  • A feedforward network computes $f(x) = W_L \sigma(W_{L-1} \cdots \sigma(W_1 x + b_1) \cdots + b_{L-1}) + b_L$.
  • Backpropagation applies the chain rule from outputs to inputs.
  • Universal approximation theorems show that neural networks can represent broad function classes.

Formulas

Layer
$$z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = \sigma(z^{[l]})$$
Loss gradient
$$\frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^\top$$
Chain rule
$$\frac{\partial L}{\partial W^{[l]}} = \frac{\partial L}{\partial a^{[L]}} \frac{\partial a^{[L]}}{\partial z^{[L]}} \cdots \frac{\partial z^{[l]}}{\partial W^{[l]}}$$

Code Example

import torch

model = torch.nn.Sequential(
    torch.nn.Linear(10, 64),
    torch.nn.ReLU(),
    torch.nn.Linear(64, 1)
)
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

# Forward + backward pass
out = model(x)
loss = loss_fn(out, y)
loss.backward()
optimizer.step()

Tags

  • deep-learning
  • chain-rule
  • universal-approximation

References

  • Deep Learning
    Ian Goodfellow, Yoshua Bengio, and Aaron Courville · MIT Press · source
  • Neural Networks and Deep Learning
    Michael Nielsen · Online book · source

Knowledge Graph