Machine Learning / DL / LLM Difficulty: Advanced

Recurrent Neural Networks

Recurrent neural networks process sequential data by maintaining hidden state. They connect to dynamical systems and have been largely superseded by Transformers for many sequence tasks.

Key Points

  • RNNs update hidden state recursively: $h_t = \sigma(W_{hh} h_{t-1} + W_{xh} x_t + b)$.
  • LSTM and GRU architectures mitigate vanishing gradients.
  • RNNs are universal approximators of dynamical systems.

Formulas

RNN hidden state
$$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$$
LSTM cell state
$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$$

Code Example

import torch.nn as nn

rnn = nn.LSTM(input_size=10, hidden_size=20, num_layers=2, batch_first=True)
out, (hn, cn) = rnn(sequence)

Applications

Tags

  • rnn
  • sequences
  • lstm

References

  • Deep Learning
    Ian Goodfellow, Yoshua Bengio, and Aaron Courville · MIT Press · source
  • Long Short-Term Memory
    Sepp Hochreiter and Jürgen Schmidhuber · Neural Computation, 1997 · source

Knowledge Graph