Tag
Layer normalization stabilizes training by normalizing each example on its own, using the mean and standard deviation of its own activations. Unlike batch normalization, it does not depend on the batch size, which is why transformers adopted it. RMSNorm is its lighter, most widely used variant today in large language models.
Batch normalization is a technique that normalises each layer's activations using the mean and variance of the mini-batch, then rescales them with two learnable parameters, gamma and beta. Introduced in 2015, it enables higher learning rates, speeds up training and stabilises deep neural networks during optimisation.
The SELU (Scaled Exponential Linear Unit) function is defined as SELU(x) equal to lambda times ELU(x), with lambda near 1.0507 and alpha near 1.6733. Those two constants make activations converge on their own towards zero mean and unit variance layer after layer, building deep networks that normalize themselves without batch normalization.
