overfitting

L1 and L2 Regularization (Weight Decay)

L1 and L2 regularization adds a term to the loss function that penalises large weights. L2, or weight decay, shrinks the weights smoothly towards zero; L1 drives them exactly to zero and yields sparse models. Both curb overfitting and improve the generalisation ability of a neural network on unseen data.