Tag
Batch normalization is a technique that normalises each layer's activations using the mean and variance of the mini-batch, then rescales them with two learnable parameters, gamma and beta. Introduced in 2015, it enables higher learning rates, speeds up training and stabilises deep neural networks during optimisation.
A learning rate schedule changes the value of η over the course of training instead of keeping it fixed. It starts with a warmup that raises η from near zero, holds a peak and then lowers it with step, exponential or cosine decay so the network converges faster and with far less oscillation.
Batch gradient descent uses all the data at each step, the stochastic version uses a single sample and mini-batch picks an intermediate group, usually 32 to 256 examples. This guide compares the three variants, their cost, their noise and why mini-batch has become the standard for training modern networks.
The learning rate is the hyperparameter that sets the size of each step when adjusting the weights during training. Too high a value makes the loss diverge; too low a value makes learning painfully slow. Typical values range from 0.001 with Adam to 0.1 with classic gradient descent.
A loss function measures how wrong a neural network is on a single example, comparing its prediction with the correct value. The cost function averages that loss across the whole dataset. That single number is exactly what training tries to reduce, step by step, using gradient descent to adjust every weight.
Direct Preference Optimization (DPO) and its variants, IPO, KTO, and SimPO, have displaced RLHF as the preferred alignment method for language models: they drop the separate reward model, cut training cost, and are easier to reproduce. RLHF still has an edge only for frontier models with very large budgets.
Synthetic data has moved from a precarious substitute for real data to a central component of modern model training: the most reliable pattern expands a real core of 500 examples with thousands of synthetic paraphrases, provided you validate diversity, correctness, and distribution, and keep at least 30% real data to avoid model collapse.
