3.3 Sequence Models: RNNs, LSTMs & Temporal Processing

3.3 Sequence Models: RNNs, LSTMs & Temporal Processing

While feedforward neural networks and convolutional networks excel at processing static, fixed-dimensional inputs like tabular records or spatial images, many enterprise artificial intelligence applications require analyzing ordered streams of data. In financial transactions, IoT telemetry, acoustic audio streams, and natural language text, individual observations cannot be understood in isolation—their semantic meaning depends fundamentally on what occurred before. Sequence models are neural architectures specialized in tracking state and temporal dependencies over variable-length ordered sequences. For the OCI AI Foundations exam, candidates must understand why standard feedforward networks fail on sequential data, how Recurrent Neural Networks (RNNs) introduce internal recurrence, why the vanishing gradient problem impairs long-term memory, and how gated architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs) resolve these limitations.

---

The Nature of Sequential and Temporal Data

Standard machine learning algorithms operate on the foundational assumption that data instances are independent and identically distributed (i.i.d.). Under the i.i.d. assumption, the model assumes that sample $n$ has no bearing on sample $n+1$, and that shuffling the order of training records does not distort underlying patterns. In sequential data, this assumption collapses completely.

Sequential Data Characteristics

1. Temporal Context: An individual element's meaning is dictated by historical context. In natural language, the word "bank" has entirely different meanings in the sequences "sat on the river bank" versus "deposited money in the bank."
2. Variable Sequence Lengths: Unlike a fixed tabular dataset with an unchanging number of columns, sequential inputs vary dramatically in length. One text prompt might contain 4 words, while the next contains 250 words.
3. Long-Range Dependencies: Events separated by substantial temporal intervals may be intrinsically linked. For example, a pronoun at the end of a paragraph ("she") may refer to a named subject introduced in the opening sentence.

Enterprise Sequence Domains

• Natural Language Processing (NLP): Machine translation, sentiment classification, entity extraction, text summarization, and dialogue generation.
• Audio and Speech Processing: Converting continuous acoustic sound wave signals into transcribed phonetic tokens (the foundation of OCI Speech).
• Financial & Market Time-Series: Stock valuation changes, algorithmic trading signals, currency volatility forecasting, and transaction fraud detection.
• Industrial IoT & Telemetry: Sensor readings monitoring vibration, temperature, and pressure variations in manufacturing machinery over time to forecast equipment failures.

Why Standard Feedforward Networks (MLPs) Fail on Sequences

Attempting to model sequential text or time-series data using standard MLPs fails for three critical reasons:

• Fixed Input Sizing: MLPs require a fixed, static input dimension. Accommodating variable-length sequences requires either truncating long sentences or arbitrarily zero-padding short sequences.
• No Memory Retention: MLPs possess no internal feedback loops or state. Once an input sample propagates through the network to produce an output, all internal activations are discarded before the next sample arrives.
• Parameter Redundancy Across Positions: An MLP must learn independent weights for every single position in a sentence. A keyword appearing at position 1 triggers entirely different weights than if that same keyword appears at position 10, preventing the model from generalizing temporal patterns across variable sequence positions.

---

Recurrent Neural Networks (RNNs)

Recurrent Neural Networks (RNNs) introduce internal recurrence—cyclical feedback connections that allow signals to persist across discrete time steps. Instead of processing an entire sequence as a single monolithic block, an RNN processes sequential elements step-by-step, maintaining an internal continuous memory called the hidden state ($h_t$).

                 h_t
                  ^ 
                  | 
[x_t] ──> [RNN Cell (W_hh, W_xh)] ──> h_t ──> [Output y_t]
                  | 
                  v 
                h_(t+1)

The Mathematical Formulation of an RNN Cell

At each discrete time step $t$, the RNN cell receives two distinct inputs:

1. The current sequence token or feature vector: $x_t$
2. The hidden state emitted from the immediately preceding time step: $h_{t-1}$

The cell computes a new hidden state $h_t$ by taking a linear combination of both inputs, adding a bias term, and passing the result through a non-linear activation function (conventionally tanh):

$h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$

If the task requires producing an output at time step $t$, the network computes a prediction vector $\hat{y}_t$ (often utilizing a Softmax layer for classification):

$\hat{y}_t = \text{Softmax}(W_{hy} h_t + b_y)$

Parameter Sharing Across Time

A defining strength of the RNN is that the weight matrices—$W_{xh}$ (input-to-hidden), $W_{hh}$ (hidden-to-hidden), and $W_{hy}$ (hidden-to-output)—are shared across every time step. The identical parameter tensors are reused whether the sequence contains 5 tokens or 500 tokens. This parameter sharing prevents parameter explosion and allows the network to process arbitrarily long inputs while recognizing patterns regardless of their temporal positioning.

Architectural Configurations

---

The Vanishing and Exploding Gradient Problem in RNNs

Although simple RNNs are theoretically capable of retaining information across indefinite time horizons, in practice they struggle to learn dependencies spanning more than approximately 10 to 15 time steps.

Backpropagation Through Time (BPTT)

To train an RNN, the network is unrolled conceptually across all $T$ time steps, creating an unrolled feedforward graph. The network computes the cumulative loss across time steps and updates parameters using Backpropagation Through Time (BPTT). BPTT propagates error gradients backwards from the final time step through each preceding time step to the beginning of the sequence.

Step t=1            Step t=2            Step t=3            Step t=T
[h_0] ──> [Cell] ──> [h_1] ──> [Cell] ──> [h_2] ──> [Cell] ──> [h_(T-1)] ──> [Cell] ──> [Loss L_T]
           ^                    ^                    ^                          ^
          x_1                  x_2                  x_3                        x_T
   <=========================== Gradient Flow Backwards Through Time =============================

The Cause of Vanishing Gradients

When computing the gradient of the loss at step $T$ with respect to the initial hidden state $h_0$, the chain rule requires calculating a continuous product of Jacobian matrices across every intermediate temporal step:

$\frac{\partial L_T}{\partial h_0} = \frac{\partial L_T}{\partial h_T} \prod_{k=1}^T \frac{\partial h_k}{\partial h_{k-1}}$

Notice the derivative $\frac{\partial h_k}{\partial h_{k-1}}$ contains the recurrent weight matrix $W_{hh}^T$ scaled by the derivative of the tanh activation function. Because the derivative of tanh is strictly bounded between $0$ and $1$ (specifically, $\tanh'(z) = 1 - \tanh^2(z) \le 1$), and if the largest eigenvalue of the recurrent weight matrix $W_{hh}$ is less than 1, multiplying these fractional values repeatedly across dozens of time steps causes the gradient to decay exponentially toward zero.

By the time the error signal reaches early time steps, the gradient has vanished. Consequently, the network cannot update its weights to reflect long-term historical context. If an RNN reads, "The clouds that gathered across the mountains all afternoon suddenly released heavy [rain]," it cannot connect "clouds" to "rain" if separated by dozens of intervening words.

Exploding Gradients and Gradient Clipping

Conversely, if the eigenvalues of $W_{hh}$ exceed 1, the gradient product can grow exponentially, resulting in exploding gradients. This manifests as numerical overflow (NaN errors), wild parameter oscillations, and model collapse. Exploding gradients are routinely managed through Gradient Clipping—scaling down the gradient vector whenever its Euclidean norm exceeds a predefined threshold:

$\mathbf{g} \leftarrow \mathbf{g} \times \frac{\text{threshold}}{\|\mathbf{g}\|} \quad \text{if } \|\mathbf{g}\| > \text{threshold}$

---

Advanced Gated Architectures: LSTM and GRU

To overcome the fundamental limitation of vanishing gradients, researchers designed gated architectures equipped with specialized mathematical conduits that regulate what information to remember, update, and forget.

Long Short-Term Memory (LSTM) Networks

Invented by Sepp Hochreiter and Jürgen Schmidhuber in 1997, the Long Short-Term Memory (LSTM) network explicitly resolves vanishing gradients by introducing a dedicated long-term memory conduit called the Cell State ($C_t$) alongside three regulatory gates.

The Cell State ($C_t$): The Information Conveyor Belt

The cell state runs horizontally across the top of the unrolled LSTM network with minimal linear interactions. Because updates to the cell state are primarily additive rather than purely multiplicative, error gradients can flow backwards across hundreds of time steps without decaying exponentially.

The Three Regulatory Gates

Each gate consists of a sigmoid ($\sigma$) neural layer that outputs values between $0$ and $1$, where $0$ represents "completely block / discard this information" and $1$ represents "completely allow / retain this information."

1. Forget Gate:     f_t = σ(W_f · [h_(t-1), x_t] + b_f)       ──> Discards irrelevant history
2. Input Gate:      i_t = σ(W_i · [h_(t-1), x_t] + b_i)       ──> Determines what new data to add
   Candidate State: C̃_t = tanh(W_c · [h_(t-1), x_t] + b_c)    ──> Proposes new candidate values
   Cell Update:     C_t = f_t * C_(t-1) + i_t * C̃_t           ──> Updates long-term conveyor belt
3. Output Gate:     o_t = σ(W_o · [h_(t-1), x_t] + b_o)       ──> Filters cell state to emit
   Hidden State:    h_t = o_t * tanh(C_t)                     ──> Emits short-term hidden state

1. Forget Gate ($f_t$): Decides what information to purge from the previous cell state $C_{t-1}$. Ingests the previous hidden state $h_{t-1}$ and current input $x_t$, outputting a filter vector between 0 and 1: $f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f)$
2. Input Gate ($i_t$) and Candidate Cell State ($\tilde{C}_t$): Decides what new information to write into the cell state. The input gate ($i_t$) identifies which values to update, while a tanh layer creates candidate values ($\tilde{C}_t$): $i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i)$ $\tilde{C}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c)$
3. Updating the Cell State ($C_t$): The new cell state is computed by element-wise multiplying the old state by the forget vector ($f_t * C_{t-1}$) and adding the scaled candidate information ($i_t * \tilde{C}_t$): $C_t = f_t * C_{t-1} + i_t * \tilde{C}_t$
4. Output Gate ($o_t$) and Hidden State ($h_t$): Decides what information from the cell state should be emitted as the updated hidden state $h_t$. The output gate evaluates the inputs, and the cell state is squashed through tanh before element-wise multiplication: $o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o)$ $h_t = o_t * \tanh(C_t)$

Gated Recurrent Unit (GRU)

Introduced by Kyunghyun Cho et al. in 2014, the Gated Recurrent Unit (GRU) is a simplified variant of the LSTM. The GRU combines the cell state and hidden state into a single unified hidden state ($h_t$) and reduces the three LSTM gates to just two gates:

1. Reset Gate ($r_t$): Determines how much of the past hidden state to forget when calculating new candidate activations.
2. Update Gate ($z_t$): Acts simultaneously as both the forget and input gates, dictating how much of the historical state to retain versus how much of the candidate state to incorporate.

Because GRUs have fewer parameters (approximately $25%$ fewer weights than an equivalent LSTM), they train faster, require less memory, and perform comparably on small to medium-sized sequential datasets.

Comparative Overview: RNN vs. LSTM vs. GRU

---

Sequence-to-Sequence (Seq2Seq) Models & The Attention Imperative

For complex language tasks where input sequences and output sequences differ in length (such as translating English to German or summarizing long articles), researchers introduced the Sequence-to-Sequence (Seq2Seq) framework (Sutskever et al., 2014):

[Encoder: RNN/LSTM]                                    [Decoder: RNN/LSTM]
"The" ──> "cat" ──> "sat" ──> [Final Hidden State h_T] ──> "Le" ──> "chat" ──> "s'est" ──> "assis"
                               (Context Vector c)

1. Encoder: A recurrent network processes the source sequence token-by-token, compiling historical information into its final hidden state, known as the context vector ($c$).
2. Decoder: A second recurrent network ingests the context vector as its initial state and generates the target sequence token-by-token.

The Information Bottleneck Problem

While revolutionary, Seq2Seq architectures suffered from a critical flaw: the fixed-length context vector bottleneck. Forcing an entire 50-word complex sentence into a single, fixed-size numerical vector (e.g., 512 floating-point values) caused severe information loss. The encoder inevitably forgot details from the beginning of the sentence by the time it finished reading the end.

This fundamental bottleneck inspired Bahdanau et al. in 2014 to invent the Attention Mechanism. Instead of relying on a single static context vector, attention allows the decoder to dynamically query and weight all intermediate encoder hidden states at each step of output generation. This breakthrough formed the direct foundation for the Transformer architecture, which completely eliminated recurrent loops in favor of self-attention mechanisms.