CS546 Learning Notes: Connecting Gradient Clipping to LSTM and GRU Through a Parameter Update

1. Gradient Explosion: Why Large Gradients Can Lead Training Astray?

First, let's look at vanilla SGD without momentum or weight decay:

$w$ is the parameter, $\eta$ is the learning rate, and $g$ is the gradient. Suppose $w=1$ and $\eta=0.1$:

At the time, I calculated that the two update magnitudes differed by $99.8$, and I realized that in the second case, a single step went too far.

The gradient describes changes near the current position. Taking a sufficiently small step along the negative gradient direction usually decreases the differentiable loss; taking a large step offers no such guarantee. Therefore, a large gradient combined with a fixed learning rate may cause parameters to overstep the appropriate position, actually increasing the loss.

Why does this happen easily in RNNs? Consider a scalar example stripped of inputs and activation functions:

If the terminal gradient is $1$ and $w=3$, backpropagating through three steps gives:

After ten steps, the same multiplicative factor is $3^{10}=59049$. This computes the gradient with respect to hidden state; when computing parameter gradients, these backpropagation signals are also used, so parameter gradients may correspondingly grow large.

"Increasing time steps" refers to the backpropagation path spanning more sequence positions, not that increased training epochs necessarily cause explosion. Real vector RNNs involve Jacobian multiplication, and not all paths continuously amplify. This scalar example demonstrates one mechanism.

2. Gradient Clipping: Limit the Length, Preserve the Direction

Initially, I understood clipping as "filtering outliers." More precisely: limit the impact of excessively large gradients, not discard them. Large gradients themselves don't necessarily indicate problematic data.

Organize the gradient to be clipped into a vector $g$, and set the maximum length as $c>0$. L2 norm clipping is:

For example, $g=[3,4]$ has length $\sqrt{3^2+4^2}=5$. With a threshold of $2$:

All components are multiplied by the same positive number, so the direction remains unchanged and the length is scaled to $2$. If we directly changed it to $[2,2]$, the ratio would change from $3:4$ to $1:1$, altering the direction; that is not norm clipping.

Lecture 5, page 44 of the course introduces this method. The referenced paper by Pascanu et al. proposes using gradient norm clipping to address gradient explosion.

Why Not Simply Lower the Learning Rate?

My initial thought was: "Lowering the learning rate only suppresses early updates; it will still explode later." This needs correction: a smaller learning rate shrinks every update, not just the first few. The real difference is that it also shrinks updates from normally-sized gradients.

The above still assumes vanilla SGD. Clipping only intervenes when a threshold is exceeded, while a small learning rate uniformly shrinks all updates. The two can also be used together.

Clipping occurs after gradient computation and does not cancel the chain multiplications in backpropagation. It also cannot solve gradient vanishing: $0.00001$ is below the threshold $5$, and after clipping remains $0.00001$.

3. LSTM: Retain Old Memories While Writing New Content

Gradient vanishing is not just about "small steps." When loss from later steps propagates to earlier positions, the signal becomes too weak, making it difficult for the model to learn to use information from the distant past.

LSTM maintains a cell state $c_t$ and updates memory through gates:

$\odot$ denotes element-wise multiplication. In practice, these quantities are usually vectors, with each component having its own gate value; we'll compute one component by hand first.

Old memory is $8$, forget gate is $0.75$, retaining $6$. Candidate memory is $0.8$, input gate is $0.5$:

Here $c_t$ can exceed $1$; the candidate memory component after tanh is within $(-1,1)$. This example with $8$ assumes memory has accumulated previously.

Are Gate Values Manually Fixed?

I then asked: "How are the input gate values defined? Is the forget gate a fixed value?"

They are computed from inputs, context, and trainable parameters:

$\sigma$ is sigmoid, mapping finite input to $(0,1)$ as a passage ratio. For example, when sigmoid input is $-2, 0, 2$ respectively, outputs are approximately $0.12, 0.5, 0.88$.

Here we use column vector convention: if input dimension is $d$ and hidden dimension is $m$, then $x_t\in\mathbb R^d$, $h_{t-1},c_t,i_t,f_t\in\mathbb R^m$, $W_i\in\mathbb R^{m\times d}$, $U_i\in\mathbb R^{m\times m}$, $b_i\in\mathbb R^m$; similarly for other gates.

The same set of parameters is shared across all time steps, but since inputs and contexts differ, gate values can be different. Training updates these parameters; gate values are computed results from each forward pass.

4. Output Gate: Memory Retained Doesn't Mean It's Presented Now

LSTM also has an output gate:

I could compute $0.25\times0.8=0.2$, but then asked again: "What exactly does this $h_t$ represent?"

Now I've clarified it: $c_t$ is internal memory, $h_t$ is the current hidden state. $h_t$ can be sent to the output layer for predictions and participates in computing next step's gate values and candidate content; $c_t$ continues along the memory path. The standard LSTM's state relationships can be compared with PyTorch LSTM formula.

For example, a language model could compute $\text{logits}_t=W_yh_t+b_y$, then apply Softmax to predict the next word. $h_t$ itself is neither a word nor a probability.

When the output gate is close to $0$, the corresponding $h_t$ component is close to zero, but $c_t$ won't be directly cleared. To retain a memory, not present it for now, and not write new content, we can set the corresponding:

I also corrected a statement: "When the output gate opens, output follows the old memory." In reality, the output gate acts on the updated $c_t$. The memory just changed to $6.4$, so the ideal case with output gate $1$ corresponds to $\tanh(6.4)$, not $\tanh(6)$.

To preserve memory approximately unchanged, besides $f_t\approx1$, the new addition term should be near zero, e.g., $i_t\approx0$. Simply keeping the forget gate near $1$ could still allow continued writing of new content.

5. Why Can LSTM Alleviate Gradient Vanishing?

Along the path from old memory directly to new memory, the direct local derivative of each component is the corresponding $f_t$ component. Therefore, when five consecutive steps are $0.99$, the gradient factor along this path is:

Compared to $0.2^5=0.00032$, the signal is preserved much better. My understanding is: "$f_t$ close to $1$ better preserves old information; multiplying by $0.2$ each step gradually causes forgetting." Plus backpropagation: gradients also more easily propagate back along this direct path to earlier positions.

This "direct" is important. The complete computation graph also includes dependencies through $h_{t-1}$ from gate values and candidate content; we can't reduce the complete derivative to just forget gate multiplication. LSTM provides more favorable propagation paths, but doesn't guarantee gradients never vanish or explode.

6. GRU: Blend Old and Candidate States Using an Update Gate

GRU doesn't maintain $c_t$ separately, but updates $h_t$. This article adopts PyTorch GRU convention:

$z_t$ is the update gate, $n_t$ is the candidate state. Here, the closer $z_t$ is to $1$, the more old state is retained; the closer to $0$, the more candidate state is used.

Some materials have $z_t$ multiply the candidate state, with opposite notation. When reading, check the formula, not just the gate name.

For example, old state is $0.8$, candidate state is $0.2$, $z_t=0.75$:

At the time, I summarized it as "either retain more old state or retain more new input." Here we need to change "new input" to "candidate state": it's not the original $x_t$, but computed content that can itself contain historical information.

7. What's the Difference Between Reset Gate and Forget Gate?

The reset gate $r_t$ in GRU appears in the generation of candidate state. Continuing with PyTorch notation:

It is multiplied onto the contribution of the old state after transformation. When $r_t\approx0$, the candidate content references history less; when $r_t\approx1$, this portion of historical contribution is allowed to pass through. PyTorch's reset gate placement differs from some original implementations, and this article consistently uses the version above.

The name "reset" initially led me to guess backwards, thinking that wanting to start fresh should mean close to $1$. Only by looking at the position did I realize that here the value represents the passage ratio.

I later asked: "Is its function similar to the LSTM forget gate?" Both control historical information, but at different positions:

So in terms of "directly retaining the old state," $z_t$ in this article's convention is more similar to $f_t$.

Two combinations helped me truly distinguish them: when $r_t\approx0,z_t\approx1$, the candidate content references history less, but the final state still mainly retains the old state; when $r_t\approx1,z_t\approx0$, the final state mainly adopts the candidate state, but the candidate state may still contain processed historical information. Allowing reference to history does not guarantee preserving it unchanged.

8. BPTT: When to Multiply, When to Add?

Gate parameters are learned from task loss; no need to manually annotate correct values for each gate. But does BPTT update parameters at each step every time it backpropagates?

Usually not. Parameters are shared across time steps; BPTT collects and accumulates gradient contributions, which are then updated by the optimizer.

In the Q&A, two time steps' contributions to the same parameter were $0.3$ and $-0.1$, which I mistakenly computed as the product $-0.03$. The correct result should be:

The rule remains: along the same path, local derivatives multiply; when multiple paths converge to the same parameter, contributions add.

Even with loss only from the second step, the shared parameter $w$ has different paths:

One path affects loss through $w\rightarrow h_2\rightarrow L_2$, another through $w\rightarrow h_1\rightarrow h_2\rightarrow L_2$. Compute derivatives along each path, then add them.

"Different time steps are different paths" is an introductory understanding. More precisely, parameter usages at different time steps form different paths, and each usage may affect multiple subsequent losses. The gradient contribution of a particular step must already include relevant subsequent propagation; it cannot be omitted or double-counted.

9. Optimizer: Actually Modifying Parameters After Receiving Gradients

BPTT is responsible for computing gradients; the Optimizer is responsible for using gradients to update parameters. The simplest example is still ordinary SGD:

Contributions from three time steps are $0.3,-0.1,0.4$; total gradient is $0.6$. If the parameter is $1$ and learning rate is $0.1$:

In the next round, if the gradient is $-0.4$:

Negative gradient causes the parameter to increase. The updated parameters are used in the forward computation of all time steps in the next iteration. This computation corresponds to SGD without momentum, weight decay, or other extra terms.

Momentum and Adam further utilize historical information to determine updates, but here I've only established the optimizer's responsibility and basic SGD computation without expanding these algorithms.

10. The Training Loop Connected This Round

A typical training iteration can be read straight through: clear previous gradients → compute states, gate values, and predictions in forward pass → compute task loss → BPTT computes and accumulates gradients of shared parameters → clip if necessary → Optimizer updates parameters.

In practice there are also arrangements like gradient accumulation across batches and truncated BPTT, so "only update after the entire sequence ends" shouldn't be treated as the only approach. The core division of labor remains unchanged: computing derivatives and updating parameters are separate steps.

Three points I most want to remember from this round: clipping limits overly large gradients but cannot restore vanished signals; gating allows the model to learn how to retain and use history, but gate values are not parameters; during backpropagation, multiply along paths and add when summing contributions to shared parameters.

Next time I see a complex formula, I'll ask first: which path does this quantity control? Is it a state computed right now, or a parameter to be updated during training? This is more reliable than just memorizing gate names.