L17 | English F3 Student Learning Log

Persona: I'm a Form 3 (Grade 9) student and I only read English. I know basic Python (variables, for loops, lists) and school algebra up to quadratics. I learn from the official pages, the videos with captions, and the book. Kaggle notebook steps are written as a practice log that follows the official notebooks.

What I understood (in my own words)

Pass a signal through many layers and a little amplification each time makes it huge; a little shrinking makes it vanish.
Initialization means setting the weights to just the right size before training starts.
ReLU cuts off negatives, so Kaiming init makes up for it by multiplying by √2.
BatchNorm keeps pulling numbers back to a standard range during training.

Where I got stuck, and how I got unstuck

Where I got stuck	How I solved it
Variance vs standard deviation	Standard deviation is the square root of variance, so it's in the same units as the original numbers.
Why 'mean 0, standard deviation 1'?	I imagined each layer as a person passing on a message. The volume must stay normal: too loud distorts, too quiet can't be heard.

My answers to the exercises

Q1. A layer has 100 inputs. With the simple '1 / √n_in' init, what's the weight standard deviation? With Kaiming '√(2 / n_in)'?

Simple: 1 / √100 = 1 / 10 = 0.1. Kaiming: √(2 / 100) = √0.02 ≈ 0.141.

Q2. If every layer scales the signal by 1.5, roughly how much larger is it after 20 layers? What does that show?

1.5²⁰ ≈ 3325 times. Even a small boost per layer explodes over many layers, so initialization should keep each layer's scaling close to 1.

Q3. What problem does BatchNorm solve?

During training the weights keep changing, so activation distributions drift and can grow or shrink. BatchNorm re-centres each layer's activations to mean 0, standard deviation 1 using the current batch's statistics, making training more stable and allowing higher learning rates.

My Zettelkasten card

Card｜Tiny differences per layer add up to huge ones

The problem with deep networks is the 'deep' part. Scaling a signal by 1.5 per layer sounds harmless, but after 20 layers it's over three thousand times bigger. Initialization and normalization aim to keep every layer's output at mean 0 and standard deviation 1, so nothing explodes or vanishes.

Our class played Chinese whispers. The first person said 'PE moves to the hall tomorrow'; fifteen people later it was 'tomorrow is a holiday'. Each person changed it only slightly, but it ended up completely different. When the teacher made everyone write it down before passing it on, the message survived, which is a bit like BatchNorm correcting at every layer. I still don't fully get why BatchNorm allows bigger learning rates, so that's next on my list.

My next step: work out 1.1, 1.5 and 0.9 to the power 20 on a calculator to see how small differences compound.

🔗 Related notes to grow next: → Maths: 'Exponential growth turns small differences into huge ones' → Communication: 'Messages passed through many hands need checking' → Habits: '1% better a day adds up over a year'

Back to lesson note：L17_初始化與正規化_202611041526