CS546 Learning Notes: From Convolutional Kernels to LSTM’s Memory Conveyor Belt — Reviewing Sequence Modeling

1. Two Diagrams of the Convolutional Layer Are the Same Thing Drawn Two Ways

At first, I couldn't understand the 3D diagram on page 9 and the neural network connection diagram, nor did I know their relationship to the sliding window diagram on page 8.

My current understanding is: the two diagrams depict the same operation. Page 8 shows the "sliding window perspective": a 2×2 kernel slides over a 4×4 input, and each time it stops, we multiply and sum element-wise, getting one number; finally we get a 3×3 feature map. Page 9 shows the "neural network perspective": we treat each output as a neuron and see which inputs it connects to.

There are two key points in the neural connection diagram:

• Each output neuron connects only to a few adjacent inputs, and it connects to all channels (depth) at that position, which is called local connectivity.
• Neurons at different positions have connection patterns that look identical, just shifted by one position overall, showing that they share the same set of weights, which is called shared weight.

So a convolutional layer can be viewed as a special fully connected layer: remove most connections, keeping only local ones; then share weights at different positions. The parameters are therefore much fewer than in an MLP, and the same pattern is recognized wherever it appears. This corresponds exactly to the two motivations discussed in the earlier "beak detector" slides.

2. Input 1 Channel, Output 2 Channels: Two Different Convolution Kernels

My understanding of the $a,b,c$ diagram on page 10 is that the input has only one channel and the output has two, meaning two different convolution kernels are used. This understanding is correct.

• The blue kernel $[w_{b1}, w_{b2}]$ produces the $b$ channel: $b_1=w_{b1}a_1+w_{b2}a_2$, $b_2=w_{b1}a_2+w_{b2}a_3$
• The orange kernel $[w_{c1}, w_{c2}]$ produces the $c$ channel

$b_1$ and $b_2$ use the same pair of weights, with the window shifted forward by one position. The kernel size is 2, stride is 1, so each channel outputs $3-2+1=2$ values.

The entire layer has only 4 weights. If we switched to a fully connected layer, 3 inputs to 4 outputs would require 12 weights.

On the next page with 2 input channels, each kernel must cover both channels at once, so each kernel has $2\times2=4$ weights. We can record a general rule:

where the number of output channels is the number of convolution kernels.

3. Spatial Substructure Made Me Think of Dense Embedding, But They're Not the Same Thing

Seeing "CNNs are good at detecting spatial substructure," I made a connection to the shift from sparse vectors to dense vectors in word embedding. Looking back, this analogy was half right.

Let me correct my description of dense vectors. It's not "compressing several words into one position":

• one-hot: The vector has one dimension per vocabulary item. Each word has its own dimension, and any two distinct word vectors are orthogonal.
• dense: Each word is a low-dimensional vector, say 300 dimensions, every dimension is shared by all words. A word's meaning is scattered across various dimensions; words with similar meanings will have similar vectors.

The two solve different problems:

The common ground is generalization through sharing: embedding lets similar words share similar representations, CNN lets different positions share the same kernel.

Is CNN a downstream task of embedding?

I then asked this question. A more precise way to say it is: CNN is a type of model architecture, not a task. In text tasks, it is a layer that comes after embedding:

Word → embedding layer → CNN (convolution + pooling) → fully connected + softmax → classification result
      Transform each word to vector   Find local phrase patterns    Complete downstream task

Two more points to note. First, embedding doesn't have to be pretrained and fixed; it can also be randomly initialized and then trained end-to-end together with the CNN. Second, CNN itself doesn't depend on embedding; when processing images the input is just pixels, but in text words are discrete symbols, so they need to be transformed into vectors first.

4. Five Hyperparameters, One by One

What Does "2D" in "2D Convolution" Mean?

It means how many directions the convolution kernel can slide, not how many dimensions the input has. The kernel on page 14 slides both right and down, so it's 2D.

Two easy places to get confused:

• Although an RGB image is $3\times H\times W$ and the kernel is $3\times k\times k$, the kernel covers all channels at once in the channel direction without sliding, so it's still 2D convolution.
• Text is an $n\times d$ matrix, which looks like an image. But the kernel also covers the embedding dimension at once (slicing a word vector into pieces to slide doesn't make sense), sliding only along the word sequence direction, so it's 1D convolution, where $d$ acts as the input channels.

Memory trick: The dimensionality of convolution equals the number of directions the kernel can move; the dimension that is covered all at once is the channel dimension, and doesn't count.

Does Kernel Size = 3 Mean the Kernel is Always Square?

Not necessarily. kernel_size=3 is just shorthand for $3\times3$ because square kernels are most common.

nn.Conv2d(1, 1, kernel_size=3)       # 3×3
nn.Conv2d(1, 1, kernel_size=(3, 5))  # 3×5, rectangular

The most typical scenario for rectangular kernels is using 2D convolution to process an $n\times d$ sentence matrix: the kernel is $k\times d$, say $3\times300$, which is a thin strip that can only slide up and down.

What this page really wants us to compare is: same 4×4 input, 2×2 kernel outputs $3\times3$, 3×3 kernel outputs $2\times2$. The larger the kernel, the larger the receptive field of each output (longer n-grams in text), the smaller the output, and the more parameters.

Is Padding to Make Output the Same Size as Input?

Yes, that's the main purpose. The example on page 18 is 5 inputs, kernel width 3:

• padding = 0: $5-3+1=3$ outputs
• padding = 1: pad one 0 on each side, getting $(5-3+2)/1+1=5$ outputs

With stride 1, taking $P=(K-1)/2$ makes the output equal in length to the input, which is "same" padding.

Two other purposes: edge elements get covered by more windows, words at the beginning and end of a sentence aren't disadvantaged; each layer doesn't keep shrinking, so we can stack many layers.

Same page with stride = 2: $(5-3)/2+1=2$, the window jumps two positions each time.

Dilation: I Got the Size Backwards

I initially thought it meant "the original size = 3 kernel becomes size = 2". Actually it's the opposite: the kernel was originally 2×2, dilation expands it, so it looks like 3×3.

In the 3×3 grid of the kernel on page 19, only the 4 corners are gray, those are the true 4 weights, the white grid in the middle is empty space. The actual receptive field is:

So the output is $4-3+1=2$, which is $2\times2$.

Its purpose is to see a larger range with the same number of parameters. By setting dilation to 1, 2, 4, 8 across layers, each layer's receptive field doubles, and the receptive field grows exponentially with layers; WaveNet does this.

Output Size General Formula

When $d=1$ it reduces to the common form $\lfloor (W-K+2P)/S\rfloor+1$. Padding makes the output larger, kernel, stride, and dilation make the output smaller.

5. Example computation: filter is a small cube

I initially didn't know what "filter" on page 20 referred to. A filter is a convolutional kernel, with size $3\times3\times3$ in this case: the first two 3's are the window size, and the last 3 must equal the number of input channels.

• The input is originally $5\times5\times3$, and becomes $7\times7\times3$ after padding = 1
• There are two filters, W0 and W1, each with a bias
• stride = 2, so the output is $(5-3+2)/2+1=3$, for a total of $3\times3\times2$

The top-left output 0 is computed as follows: W0 is placed at the top-left corner, each of the three channels multiplies element-wise and sums, then these three results are added together, and finally the bias is added.

The key point here is: when a filter stays at one position, it outputs only one number, and the results from three channels are added together. With several filters, the output will have that many channels. The parameter count is $2\times(27+1)=56$.

I verified all 18 outputs cell by cell with these numbers, and they all match the course slides.

The connection diagram on the right also matches up: the three rows of circles at the bottom are the 3 input channels, with white circles at both ends representing padded 0's; the cyan and orange sets of lines are the two filters; the output neurons are spaced two cells apart, which is stride = 2.

6. Are convolutional kernels fixed?

It depends on what you're comparing to.

• Within one forward pass, they are fixed. The same filter uses the same set of numbers wherever it slides—this is weight sharing.
• During training, they are not fixed. The numbers in the kernel are learnable parameters: they are randomly initialized and then updated through backpropagation and gradient descent. The numbers in the course slides are just a snapshot at one moment in time.

The example above demonstrates forward propagation. During backpropagation, there's a detail: a certain weight in W0 is used at 9 positions, so its gradient is the sum of gradients from those 9 positions. RNN's BPTT follows the same logic: a single $W$ is shared across all time steps, and gradients must be summed across all time steps.

This is also the difference between CNN and traditional image processing: edge detection kernels like Sobel are manually written and fixed, while CNN kernels are learned from data.

7. What's the relationship to convolution in mathematics?

Convolution in mathematics, in discrete form, is:

If we view $f$ as the input, $g$ as the kernel, and $i$ as the output position, it's "multiply corresponding elements and sum." Note that $g$ is indexed by $i-a$: as $a$ moves forward, $g$'s index moves backward, equivalent to flipping the kernel first, then sliding.

What CNN actually computes is:

No flipping—mathematically called cross-correlation. We still call it convolution because the kernel is learned, and flipping or not flipping just means the model learns a left-right flipped kernel, with the same effect.

8. RNN's W, U, V, and the dimensionality of $s_t$

Suppose word embeddings are 300-dimensional, hidden size is 128, and the vocabulary has 10000 words:

• $U$ is $128\times300$, $W$ is $128\times128$, $V$ is $10000\times128$
• $Ws_{t-1}$ and $Ux_t$ are both 128-dimensional, and after adding and passing through σ (element-wise, dimension unchanged), $s_t$ is 128-dimensional
• $Vs_t$ is 10000-dimensional, and after softmax becomes a probability distribution over the vocabulary

128 is a human-chosen hyperparameter. Since $s_t$ must multiply with $W$ to pass further, $W$ must be a square matrix.

In the unfolded diagram, each step shows the same $U$, $W$, $V$—this is the same idea as weight sharing in CNN: CNN shares weights across positions, RNN shares weights across time steps, so the parameter count doesn't increase with sequence length.

9. What does the probability formula in RNN-LM mean?

It means: given the first t words, the probability that the next word is the j-th word in the vocabulary equals the j-th component of the output vector $\hat y_t$.

For example, after reading "wreck a", $\hat y_2$ is a 10000-dimensional vector. If "nice" is the 537-th word, then $P(\text{nice}\mid\text{wreck, a})=\hat y_{2,537}$. All components sum to 1, which is guaranteed by softmax.

The conditional part shows the entire history $x_1,\dots,x_t$, because $h_t$ is recursively computed and in theory captures all context from the first word to now. This differs from n-grams, which only look at the previous $n-1$ words. But "in theory" is important—in practice, information from too far back is forgotten due to vanishing gradients.

During training, if the actual next word is $w_{537}$, the loss for this step is $-\log\hat y_{t,537}$. The loss for the entire sentence is the sum of losses across all steps.

10. The Green Box in Backprop: How Error Propagates Backwards

Page 38 decomposes the gradient of a weight into two parts:

So the gradient equals the error at the endpoint of this edge times the activation at the starting point. The red box half is saved during forward propagation, and the difficulty lies in computing the green box's $\delta$.

Last Layer

$z_i^L$ only affects loss through $a_i^L$:

Backward Recursion

$z_i^l$ flows through $a_i^l$ to all neurons in the next layer, creating multiple paths that must be summed by the chain rule:

$\sum_k w_{ki}\delta_k$ is the dot product of the $i$-th column of $W$ with $\delta$, which is exactly the $i$-th component of $W^T\delta$:

During forward pass, $W$ sends signals from layer $l$ to layer $l+1$; during backward pass, error travels back along the same edges, so we use $W^T$.

Why Vanishing and Exploding Gradients Occur Here

Assume each layer has only one neuron, expanding all the way:

The gradient in the first layer equals the error in the last layer times a chain of factors. With 10 layers:

The $a_j^{l-1}$ in the red box is multiplied only once, but the $\delta$ in the green box is multiplied again with each backward layer, so the problem primarily occurs here.

RNNs suffer even more severely because after unrolling, the same $W$ is used at every step. Repeatedly multiplying the same matrix over dozens or hundreds of times is dominated by its largest eigenvalue: if less than 1, it approaches 0; if greater than 1, it diverges.

11. Pages 49–51: Why Does Hidden State Need Gates?

These three pages provide motivation before LSTM, using three linguistic phenomena to show why ordinary RNNs are insufficient: they mix new input into the hidden state at every step.

• Long distance dependencies: The dog that was chasing the cat is followed by barked, not purred. The most recent word is cat, but the true subject is dog at the beginning, separated by a subordinate clause. The model needs to remember dog throughout.
• Irrelevant segments: I would like to and I need to are irrelevant to determining intent; what determines the class is set an alarm later. The model needs to skip irrelevant words and prevent them from being written into memory.
• Discontinuous segments: A conversation discusses homework and embeddings, then has an aside about COVID. The model needs to store the original topic during the interruption, then resume it, finally outputting only relevant parts.

Page 52 summarizes these as: hidden state needs specialized mechanisms that decide when to update, when to reset, when to use. LSTM uses three gates (input, forget, output) plus a memory cell to do this; GRU uses two gates (update and reset). (The "memory call" in the lecture notes should be a typo for memory cell.)

The general form of a gate is:

Then it is element-wise multiplied onto the vector being controlled. The sigmoid output is between 0 and 1, interpretable as a pass-through ratio, and can differ at each time step and each dimension.

12. What Is That Big A? Open It Up

The A on page 53 is the RNN cell, the block of computation executed at each time step. The upper half shows the folded and unfolded diagrams; after unfolding, each A is the same A with identical parameters.

The lower half opens up the central A: $h_{t-1}$ enters from the left, $x_t$ enters from below, they are concatenated into $[h_{t-1},x_t]$, pass through a yellow tanh layer, and the output is duplicated—one copy goes up as the output, the other goes right to the next step.

This is completely equivalent to $\tanh(Ws_{t-1}+Ux_t)$: multiplying the concatenation by a large matrix is the same as having that large matrix be $[W\mid U]$.

Remember the diagram legend: yellow blocks are parameterized neural network layers, pink circles are parameter-free element-wise operations.

When opening up LSTM's A, it contains 4 yellow layers (3 sigmoids and 1 tanh), and the lines between cells increase from 1 to 2: the upper line is cell state $C$, the lower line is hidden state $h$.

13. Cell state, candidate values and output gate

Cell state: conveyor belt for long-term memory

Cell state $C_t$ is a vector with the same dimension as $h_t$, which is the line running through the top of the diagram. It goes from $C_{t-1}$ to $C_t$, encountering only one × and one +:

No weight matrices, no nonlinear functions—this is what the lecture notes call "minor linear interactions". Once something is put on, if the gates don't touch it, it travels forward completely unchanged.

During backpropagation $\partial C_t/\partial C_{t-1}\approx f_t$, so as long as the forget gate is close to 1, the gradient doesn't need to be multiplied repeatedly by $W$ and the derivative of tanh.

What is $\tilde C_t$ with the tilde?

It's called candidate cell state, the new content prepared to be written at this step. The tilde means "draft"—how much gets written is decided by $i_t$.

The content uses tanh because it has both positive and negative values, allowing each dimension to be adjusted up or down; the gate uses sigmoid because 0 to 1 is exactly a proportion.

The lecture notes mark "Vanilla RNN" next to $\tilde C_t$ because its formula is identical to ordinary RNN. LSTM didn't discard RNN computation; it demoted it to "making suggestions": ordinary RNN's result directly overwrites old memory, while LSTM's is just a candidate that must pass through $i_t$ filtering, then be added to old memory that has been filtered by $f_t$.

Computing one step through

• $C_{t-1}=[0.9,\ -0.5,\ 0.3]$, $f_t=[1,\ 0,\ 1]$, giving $[0.9,\ 0,\ 0.3]$
• $\tilde C_t=[0.8,\ -0.9,\ 0.1]$, $i_t=[0,\ 1,\ 0]$, giving $[0,\ -0.9,\ 0]$
• Sum: $C_t=[0.9,\ -0.9,\ 0.3]$

Dimensions 1 and 3 stay unchanged, dimension 2 switches from information about the old subject to information about the new subject. This is what the orange-text example in the lecture notes means.

It's not permanent either. If $f_t$ keeps being 0.9, after 50 steps only $0.9^{50}\approx0.5\%$ remains. How long it can remember is determined by the learned forget gate.

Output gate: reading from memory what's needed for this step

The tanh here is the pink oval—no parameters, just compresses $C_t$ to $(-1,1)$. Continuing the example above, $\tanh(C_t)\approx[0.72,\ -0.72,\ 0.29]$, and if $o_t=[0,1,0]$, then $h_t=[0,\ -0.72,\ 0]$.

The lecture's example: after reading the subject, the next word is likely a verb, at which point we need to know whether the subject is singular or plural—the output gate opens these dimensions. The key point is that the output gate only affects reading, not the memory itself; information not read out at this step can still be read later.

Is $h_t$ short-term memory?

You can think of it that way: $C_t$ is long-term memory, $h_t$ is short-term memory, or working memory. It's completely recomputed at each step and only contains the portion released by the output gate. By analogy: $C_t$ is a notebook, $h_t$ is the page opened at this step.

But "short-term" means $h_t$ cannot store things on its own, not that it only contains recent information. Content stored in $C$ at the first word will appear in $h_{20}$ if the output gate opens at step 20.

Also, let me correct my own terminology: $h_t$ is the hidden state, a vector, not a hidden layer. A layer refers to the entire cell.

LSTM's name comes from this: Long Short-Term Memory—it extends the short-term memory of RNN.

Is the final softmax also $Vh_t$?

Yes. The output layer is not part of the LSTM cell; it sits on the outside, the same as ordinary RNN:

LSTM only replaced "how to compute $h_t$". The output uses $h_t$, not $C_t$. Which time steps connect to the output layer depends on the task:

The 4 yellow layers each have a set of $W$ and $b$, so LSTM's parameter count is roughly 4 times that of ordinary RNN.

14. Two LSTM variants: one adds connections, one removes connections

• Peephole connections: standard LSTM gates only see $[h_{t-1},x_t]$, not the cell state itself; when the output gate is closed, the gates know nothing about what's stored in memory. Peephole lets gates directly peek at $C$, for example $f_t=\sigma(W_f[C_{t-1},h_{t-1},x_t]+b_f)$, suitable for tasks requiring timing or counting.
• Coupled input and forget gates: set $i_t=1-f_t$, forget only when new content is written, replacing the forgotten portion with new input—one fewer gate.

The second variant is exactly the bridge to GRU: GRU's update gate uses this $z$ and $1-z$ coupled approach.

15. GRU's reset gate: close to 0 is not the same as forgetting old memory

I asked: does the reset gate being closer to 0 mean less dependence on previous memory? The direction is right, but there's a qualification.

$r_t$ only appears in the candidate value $\tilde h_t$. When $r_t\approx0$, the candidate only looks at the current input, equivalent to a feedforward network; when $r_t\approx1$, it's ordinary RNN.

How much old memory remains in the end is decided by $z_t$. Even if $r_t=0$, the $(1-z_t)\odot h_{t-1}$ in the second equation can still carry old memory through unchanged.

Comparing to LSTM: $z_t$ corresponds to $i_t$, $1-z_t$ corresponds to $f_t$; GRU has no output gate, no separate cell state either, $h_t$ serves as both memory and output; 3 sets of weights versus LSTM's 4, so it's lighter.

Note for exam: some textbooks write $z$'s direction the opposite way. According to the lecture formula, $z_t=0$ means keeping old memory.

16. Final Page: LSTM, Transformer, and S4

LSTM is slow because it can only compute sequentially. $h_t$ depends on $h_{t-1}$, and the next step cannot begin until the previous one finishes, so GPUs' parallel capabilities cannot be fully utilized.

Transformer can parallelize, at the cost of quadratic complexity. Attention allows each word to compute relationships directly with all other words, and an entire sentence can be computed in one matrix multiplication; however, computing an $n\times n$ matrix means that doubling the length multiplies the cost by 4. Additionally, any two words are only one step apart, making it easier to learn long-distance dependencies.

S4 tries to get the best of both worlds. The intuition behind LSTM is to carry the past in a state vector as you move forward—a good idea, but the problem is that it's sequential. S4 retains the state recursion but makes it linear:

Linearity means we can expand directly:

This is a convolution between the input and a fixed kernel $K=(CB,\ CAB,\ CA^2B,\dots)$. During training, treat it as long convolution and compute in parallel; during inference, switch back to recursion, where each step costs a constant, suited for streaming tasks. "Structured" means $A$ is initialized in a special way that can compress very long history. It's interesting that convolution from the beginning of this lecture appears again here.

17. The Few Statements I Most Need to Remember

• A convolutional layer is a fully connected layer with most connections removed and weights shared.
• The depth of one filter equals the number of input channels; covering one position outputs a single number; the number of filters is the number of output channels.
• The dimensionality of convolution depends on how many directions the kernel slides; what gets covered all at once is the channels. Text CNNs typically use 1D convolution.
• Dilation expands a small kernel; it doesn't shrink a large kernel.
• The kernel is fixed in one forward pass and updated during training; gradients from shared weights must be accumulated.
• The RNN's $W$ is a square matrix; $\hat y_{t,j}$ is the probability that the next word is the j-th word.
• Gradient = $\delta$ at the endpoint × $a$ at the starting point; $\delta$ multiplies by $W^T$ and $\sigma'$ as it goes back one layer, which is the source of vanishing and exploding gradients.
• The cell state line only has × and +; $\tilde C_t$ is a downgraded ordinary RNN; the output gate only reads, it doesn't change memory.
• The GRU's $r_t$ only affects the candidate value, and how much old memory to keep is determined by $z_t$.

After matching these against the slides, when I look at the diagrams again, I first ask two questions: does this block have parameters? is the line transmitting memory, output, or gate values? This way, the diagrams become less intimidating.

Materials and Corresponding Lecture Slides

This article is primarily based on a review chat from Lecture 5 and Lecture5_SequenceModeling.pdf, reorganized around actual questions asked.

• CNN motivation and two ways to draw convolutional layers: PDF pages 3–13.
• Hyperparameters (dimensionality, kernel size, stride, padding, dilation): pages 14–19.
• Example computation: pages 20–23, examples from cs231n Convolutional Networks.
• Relationship to mathematical convolution: pages 24–28; pooling: pages 29–31.
• RNN definition, RNN-LM, Backpropagation, BPTT: pages 33–42.
• Gradient problems and clipping: pages 43–45.
• Why gates are needed: pages 49–52; LSTM: pages 53–60; LSTM variants: pages 61–63.
• GRU: pages 65–68; Pros/Cons and discussion of Transformer and S4: pages 70–72.
• S4 original paper: Gu et al., Efficiently Modeling Long Sequences with Structured State Spaces, ICLR 2022.