L18 | English F3 Student Learning Log

Persona: I'm a Form 3 (Grade 9) student and I only read English. I know basic Python (variables, for loops, lists) and school algebra up to quadratics. I learn from the official pages, the videos with captions, and the book. Kaggle notebook steps are written as a practice log that follows the official notebooks.

What I understood (in my own words)

Momentum is like a ball rolling downhill: it has inertia, so a small bump doesn't change its direction.
Adam is the most-used optimizer today; it combines two good ideas.
The learning rate can change during training: big steps at first, small steps near the end.
ResNet layers only learn 'how much to change', so they can be stacked very deep.

Where I got stuck, and how I got unstuck

Where I got stuck	How I solved it
The moving-average formula	I filled a spreadsheet column and saw m slowly catching up with g instead of jumping.
Why x + f(x) helps	It's like editing an essay: you don't rewrite it, you mark changes on the original. The original is x, the edits are f(x).

My answers to the exercises

Q1. With the moving average m = 0.9 × m + 0.1 × g: m = 10 and the new gradient g = 0. What is m after the update, and what does this show about momentum?

m = 0.9 × 10 + 0.1 × 0 = 9. Even though this gradient was 0, m keeps 9, so one gradient can't suddenly flip the direction. That inertia smooths out noise.

Q2. A residual connection outputs x + f(x). Why does this make very deep networks easier to train?

If a layer can't learn anything useful, f(x) can just stay near 0 and the output equals x, so information and gradients pass straight through without being damaged. Adding depth won't make things worse than a shallower network.

Q3. Why should the learning rate get smaller late in training?

At the start you're far from the best answer, so big steps are faster. Near the bottom, big steps make you bounce around the valley. Smaller steps let you settle close to the lowest point.

My Zettelkasten card

Card｜Learning only the correction is easier than starting over

ResNet's residual connection writes the output as 'the original input plus a correction'. Each layer doesn't have to learn the whole answer from scratch, only how much to adjust; if nothing needs changing, the correction stays near 0. That's why networks can be stacked very deep and still train well.

I learned this with essays. When my English teacher returned an essay, I used to rewrite the whole thing, and I often broke sentences that were already good. Now I keep the original and only fix what's marked: spelling, clumsy sentences, weak examples. My grades went up steadily. If the essay's whole argument is wrong, though, small fixes aren't enough, and sometimes you do need to start again.

My next step: for my next essay, mark corrections in red instead of rewriting, and compare the before and after grades.

🔗 Related notes to grow next: → Writing: 'Good editing keeps what's right and fixes what's wrong' → Optimisation: 'Momentum carries you over small bumps' → Training: 'Explore with big steps, refine with small ones'

Back to lesson note：L18_加速SGD與ResNet_202611041526