Carry state through a sequence
Recurrent models process a sequence step by step while carrying a hidden state that depends on earlier steps.
The idea
Read one item at a time while carrying a small notebook summarizing what matters from earlier items.
A simple RNN updates h_t from x_t and h_{t-1}. LSTM and GRU add gates that control how information is retained, updated, or exposed.
How it works
- Read input x_t at time t.
- Combine x_t with previous state h_{t−1} using shared parameters.
- Produce new state h_t and optional output.
- Backpropagate loss through the unrolled sequence; gated architectures modify how state and gradients flow.
Simple RNN recurrence
The same input-hidden and hidden-hidden parameters are reused across time steps.
h_t = tanh(x_t W_ihᵀ + b_ih + h_{t−1} W_hhᵀ + b_hh)
Check your understanding
What makes an RNN recurrent?
- Its current hidden state depends on the previous hidden state
- It always uses images
- It never shares parameters
- It has no sequence order
Answer and explanation
The recurrent connection feeds prior hidden state into the current step, giving the model a state that evolves across the sequence.
A common misconception
LSTM and GRU do not give unlimited memory. Gates improve state control, but information can still be lost and long contexts remain challenging.
Deep Learning
Build intuition for weighted neurons, shared filters, recurrent state, and self-attention before opening the full labs.