Learning path Advanced
Stability, Initialization and Regularization
What keeps a deep network stable: weight initialization (Xavier/Glorot and He), batch and layer normalization, and dropout.
- 4 resources
- 3 views
- ~28 min
What you’ll learn on this path
This path explains what keeps a deep neural network stable during training: how to initialize weights, how to normalize activations layer by layer, and how to prevent overfitting with dropout. It is built for readers who already know backpropagation and gradient descent and want to understand why a deep network diverges, stalls, or generalizes poorly.
What you’ll be able to do
By the end you will be able to diagnose vanishing or exploding gradients, pick the right initialization for a given activation function, and decide between batch normalization and layer normalization for a specific problem. This is an advanced path: it assumes comfort with matrix algebra, gradient calculus, and the fundamentals of neural network training.
How the sequence builds
It starts with weight initialization using the Xavier/Glorot and He schemes, which fix each layer’s initial variance so the signal neither fades nor blows up during the first forward and backward passes. From there it moves to batch normalization, the technique that recalibrates activations during training and made it possible to train networks far deeper than was practical before 2015. Next it compares layer normalization and its variants, the alternative that dominates in transformers and in tasks with small batches or variable-length sequences. It closes with dropout and its mathematical interpretation, the regularization technique that randomly turns off neurons during training and can be read as a way of averaging an ensemble of subnetworks.
Weight Initialization: Xavier/Glorot and He
Weight initialization sets the starting scale of the matrix W before training begins. Xavier/Glorot, from 2010, splits the variance between inputs and outputs and suits sigmoid and tanh; He, from 2015, doubles it for ReLU, which zeroes half the activations. A poor choice stalls or breaks learning.
Batch Normalization
Batch normalization is a technique that normalises each layer's activations using the mean and variance of the mini-batch, then rescales them with two learnable parameters, gamma and beta. Introduced in 2015, it enables higher learning rates, speeds up training and stabilises deep neural networks during optimisation.
Layer Normalization and Variants
Layer normalization stabilizes training by normalizing each example on its own, using the mean and standard deviation of its own activations. Unlike batch normalization, it does not depend on the batch size, which is why transformers adopted it. RMSNorm is its lighter, most widely used variant today in large language models.
Dropout and Its Mathematical Interpretation
Dropout is a regularization technique that switches neurons off at random during training, with retention probability p, and divides the surviving activations by p to preserve their scale. At inference the full network runs without dropping anything. Mathematically it amounts to averaging a huge ensemble of subnetworks that share weights.