CS546 Learning Notes: So This Is What Parameter Updates Mean, From Linear Regression to SGD

1. Don't Look at a Long String of Formulas, Give Me a Computable Example

There is only one sample: input $x=2$, correct answer $y=5$.

The model is:

Initial parameters $w=1,b=0$, learning rate set to $0.1$. The loss we use is:

Here $\hat y$ is the prediction, $y$ is the known correct answer, and $w,b$ are parameters to be learned. During training, we change the parameters, not adjust the correct answer to match the prediction.

y-hat = 2, L = 4.5, is that right?

Yes:

I have no issues up to here. The lecture slides correspond to Lecture 2 page 50 on linear models, and page 63 on single-sample loss. The $1/2$ in front is a convention chosen here; when taking derivatives, it cancels out the factor of $2$ brought down by the square.

2. Wait, I Forgot a Bit About the Chain Rule

The next step is to find the partial derivative of $L$ with respect to $w$. My first instinct was: how should I write the first step exactly?

First, let's separate the dependencies:

$w$ first affects the predicted value, and the predicted value then affects the loss, so:

Calculate the two parts separately. When taking the partial derivative with respect to $w$, treat $x,y,b$ as constants:

Therefore:

The same logic applies to $b$, except the derivative of $wx+b$ with respect to $b$ is $1$:

The two partial derivatives together form the gradient of the parameters: $\nabla_{(w,b)}L=(-6,-3)$.

For this part, review Lecture 2 pages 76–77 on Partial Derivatives and Gradients, and Lecture 3 page 42 on the Chain Rule. Understanding the chain rule with this simple example first, then looking at the notation for multi-layer networks will be easier.

3. The Gradient is Negative, So Why Do the Parameters Get Bigger?

The rule for parameter updates is:

Thus:

Is it because the gradient indicates the direction in which loss increases, and the gradient here is negative, so to make loss decrease, we need to make $w$ and $b$ bigger?

This understanding is correct. More precisely, the gradient points in the direction of fastest loss growth near the current position, and we update by moving in the negative gradient direction. Here both partial derivatives are negative, meaning that at the current point, increasing either $w$ or $b$ slightly will decrease the loss.

We can also understand this from the prediction value: currently the prediction is $2$, the target is $5$, and $x=2>0$. Increasing either $w$ or $b$ will push the prediction upward.

Recalculate after the update:

The loss dropped from $4.5$ to $1.125$. This is one complete update: predict, calculate loss, compute gradient, update parameters, then check the new result.

The phrase here is "increase slightly." If the learning rate is too large, we might overshoot the right position, causing the loss to increase instead. Lecture 3 pages 4–11 discuss this kind of update and direction of movement.

4. So is This Entire Process Called Linear Regression?

My understanding at the time was: the prediction formula is linear, we start with a random set of $w,b$, then gradually update based on the correct answer $y$ and loss, finding the best parameters. Is this entire process called Linear Regression?

We need to separate two concepts:

• Linear Regression is a combination of model and task: use a weighted sum of inputs plus a bias to predict a numerical value.
• Gradient Descent is a training method: adjust parameters step by step through gradients.

So what we just did is accurately described as: training Linear Regression with Gradient Descent.

Linear regression can also be solved directly by taking derivatives and setting them to zero, as shown in Lecture 2 pages 55–61 for single-feature derivation. Initialization doesn't have to be random either; we can start this model from $w=0,b=0$. The goal of iterative training is to minimize loss, but a finite number of updates doesn't mean we've reached the minimum precisely.

As a side note, let me record the models and tasks asked about later:

Models include Linear Regression, Logistic Regression, Softmax Regression, MLP, Decision Tree, etc. The same task can use different models, and the same type of model can also serve different tasks. Logistic Regression, despite "regression" in its name, is typically used for classification. For now, we're just learning the names; this doesn't mean I can implement them yet.

5. The Teacher Wrote What We Just Did in One Line, and I Couldn't Understand Again

The update formula on Lecture 3 page 4 can be organized as:

Does this mean updating the parameters? How does it make me so dizzy?

After breaking it down, it doesn't contain any particularly mysterious operations beyond what we did:

Read the whole line as: the old parameters minus the learning rate times the average gradient of this group of samples, giving the new parameters.

With only one sample, the average gradient is still $(-6,-3)$, so it's:

This text treats $K$ as a set and $|K|$ as a quantity; other pages in the slides sometimes use $K$ directly to denote the number of samples. You need to look at context.

6. What's This About $K$ Samples? Why Average?

A sample is a pair of input and correct answer $(x,y)$. For example, $(2,5)$, $(1,3)$, $(3,7)$ are three samples.

Is it because they share a loss function, so we need to average?

Close, but the causality needs adjusting. Each sample uses the same loss formula, but the resulting loss values can differ. We choose to measure the model's overall performance on this group of data using the average loss:

Taking the derivative of the average loss gives the average gradient:

It's not "using the same loss formula means you must average," but rather that the objective is defined as average loss, so the resulting derivative naturally is the average gradient. Lecture 2 pages 63–64 correspond to this overall training objective.

7. I Can Do Matrix Form, But I Need to Clarify Why Parameters Are Shared

If there are three houses, each with area and age as two features, then:

I can answer this shape. Each row of $X$ is one house, each column is a feature, $b$ is a shared scalar, and $b\mathbf1$ means adding it to each prediction. The slides often simplify this to $X\mathbf w+b$.

Is it because each house corresponds to the same loss function, so we're just writing the prediction formula three times in matrix form?

The second half is right, but the reason for sharing parameters is: we want to learn a rule that applies to different houses. The input for each house varies, but the parameters remain shared, so the 4th house we've never seen can be directly plugged into the model for prediction.

Sharing model parameters and using the same loss formula are two different things. If we build a separate model for each house, we could still use the same squared error formula, but that way we wouldn't learn a common rule directly applicable to new houses.

Corresponds to Lecture 2 pages 62, 65.

8. Update with Multiple Samples, Let Me Calculate Once More

This time we use only two samples $(2,5)$ and $(1,3)$, still starting from $w=1,b=0$.

Note that both samples use the same set of old parameters, with no updates in between.

The average gradient is:

The learning rate is still $0.1$:

This is updating once using a batch composed of two samples, averaged.

9. Finally Distinguishing Batch, Step, and Epoch

With 100 samples and batch size 10, updating once per batch means one epoch is 10 steps. Training for 3 epochs means 30 updates.

The next step continues with the parameters from the previous step, and parameters are not reset when entering a new epoch. Lecture 3, pages 12–15, correspond to these concepts.

10. Can We Also Update Inside a Loop? I Thought That Would Only Optimize for Individual Samples

When looking at typical Python training loops, I noticed: updating outside the sample processing loop accumulates gradients first, then averages them. What if we update after processing each sample?

That would update once at $(2,5)$, then update again at $(1,3)$. They're each seeking the local optimum for their own sample, so the overall loss won't actually decrease?

The first part is correct, but the latter judgment is not.

First, processing each sample only takes one step in parameter space—we don't find the optimum for that sample. Second, sample-wise updates can also decrease overall loss, it just doesn't guarantee that loss decreases at every single step.

Using the same two samples, starting again from $(w,b)=(1,0)$:

1. After the first sample's update, $(w,b)=(1.6,0.3)$.
2. The second sample predicts using the new parameters: $1.6\times1+0.3=1.9$.
3. The second sample's two gradients are both $1.9-3=-1.1$.
4. Update again to get $(w,b)=(1.71,0.41)$.

The initial average loss is $(4.5+2)/2=3.25$. After two sample-wise updates:

Indeed lower than the initial value. This example shows it can improve overall performance, but not that the sample-wise method is guaranteed to always be better. Here the sample-wise method took two steps while the batch method took one step; we can't judge training efficiency based on this comparison alone.

In code, we can't simply move the update statement into the loop: sample-wise updates should use the gradient of the current sample, while batch updates should use the average gradient of the current batch. We can't just move the code without changing the gradient calculation.

11. SGD is This? I'd Never Heard This Name Before

SGD is Stochastic Gradient Descent. According to Lecture 3, pages 13–14, it updates parameters using the gradient from one sample at a time.

"Stochastic" typically refers to random sampling, or shuffling sample order before each epoch. It doesn't mean randomly deciding which direction parameters should go—the direction still comes from gradients. The fixed order in the two-sample example was just for convenience in showing sample-wise updates.

Why can a single sample provide a useful direction? Suppose at the same current parameters, three samples have gradients w.r.t. $w$ of:

The overall average is $-3$, so the average gradient suggests increasing $w$. If this step randomly draws the second sample, we'd use $+2$, decreasing $w$—possibly even opposite to the overall direction.

But with uniform random sampling at fixed parameters:

That is, the expected value of a randomly sampled gradient equals the overall average gradient. Here $N$ is the total number of samples in the training set. This explains why random gradients can estimate the overall gradient—it's not saying parameters stay constant across continuous updates, nor that every single step decreases loss. Random shuffling before sequential traversal is also common, but it's not exactly the same as independent uniform sampling at each step.

With a million samples, full-batch method must process all of them before updating once, while SGD can update after processing one. Its single update is cheap, but the direction fluctuates more; the total time to reach the same effect isn't necessarily shorter.

So when we calculated two samples together: if that's all the training data, it's called full-batch; if it's just a small group from a larger training set, it's called mini-batch. The term "mini-batch SGD" also appears; Lecture 3, page 15, uses this terminology. You need to check the batch size to determine what actually happens during updates.

12. What Part Have I Figured Out Now?

I can now hand-calculate gradients and updates for single and two samples, explain why parameters move along negative gradients, and break down the teacher's one-line average gradient formula. The relationships between batch, step, epoch, and sample-wise SGD now have actual numbers to connect them to.

But this doesn't mean I can already train neural networks. The ordinary Python training loop—I've only understood the core process so far. I still need to write, run, and modify it myself; softmax, classification loss, multi-layer networks, full backpropagation, and word vectors haven't been covered yet. The difference between training loss decreasing and whether the model can predict new data will also need follow-up.

Next time you see an update formula, you can first substitute $x=2,y=5,w=1,b=0$. When a formula doesn't make sense, ask yourself: which step in my hand calculation does this term correspond to?

Lecture Review Locations

Page numbers are counted as shown in the PDF reader, with the cover as page 1. Formulas use unified notation, and additional numerical calculations come from this learning discussion.