Tag
Layer normalization stabilizes training by normalizing each example on its own, using the mean and standard deviation of its own activations. Unlike batch normalization, it does not depend on the batch size, which is why transformers adopted it. RMSNorm is its lighter, most widely used variant today in large language models.
A learning rate schedule changes the value of η over the course of training instead of keeping it fixed. It starts with a warmup that raises η from near zero, holds a peak and then lowers it with step, exponential or cosine decay so the network converges faster and with far less oscillation.
The GELU (Gaussian Error Linear Unit) function multiplies each input by the probability that a standard normal falls below that input. The result is a smooth curve with a continuous derivative that weights inputs by their magnitude, and it has become the default activation inside BERT and GPT.
