Tag
Dropout is a regularization technique that switches neurons off at random during training, with retention probability p, and divides the surviving activations by p to preserve their scale. At inference the full network runs without dropping anything. Mathematically it amounts to averaging a huge ensemble of subnetworks that share weights.
Layer normalization stabilizes training by normalizing each example on its own, using the mean and standard deviation of its own activations. Unlike batch normalization, it does not depend on the batch size, which is why transformers adopted it. RMSNorm is its lighter, most widely used variant today in large language models.
Batch normalization is a technique that normalises each layer's activations using the mean and variance of the mini-batch, then rescales them with two learnable parameters, gamma and beta. Introduced in 2015, it enables higher learning rates, speeds up training and stabilises deep neural networks during optimisation.
Weight initialization sets the starting scale of the matrix W before training begins. Xavier/Glorot, from 2010, splits the variance between inputs and outputs and suits sigmoid and tanh; He, from 2015, doubles it for ReLU, which zeroes half the activations. A poor choice stalls or breaks learning.
Second-order methods use the Hessian, the matrix of second derivatives of the loss, to orient each step better than the gradient alone. Newton's method updates the weights by subtracting the inverse Hessian times the gradient, but inverting it is far too costly in large networks, so practitioners fall back on quasi-Newton approximations.
Nesterov accelerated gradient improves classic momentum by evaluating the gradient at a look-ahead point, the one inertia is about to reach, instead of the current point. That anticipation step corrects the trajectory before overshooting and reaches O(1/k²) convergence on smooth convex problems, the fastest a first-order method can achieve.
In large neural networks the loss surface almost never traps training in a bad local minimum. The real obstacle is saddle points, far more common in high dimensions. This article explains local versus global minima, why stochastic gradient descent slips past them, and what role convexity plays in the whole picture of optimisation.
In a dense layer with z equal to Wx plus b, the Jacobian matrix of the output with respect to the input is the weight matrix W itself. From it come the two training rules: the gradient with respect to the weights is delta times x transposed, and the gradient with respect to the input is W transposed times delta.
Backpropagation applies the chain rule to share a network's error layer by layer. You first compute the error of the output layer, propagate it backward, and use it to obtain the gradients of every weight and bias in a single pass, summarised in four compact equations you can code directly.
Backpropagation shares out the blame for the error among all the weights of a neural network. It propagates an error signal backwards layer by layer, multiplying by the local derivatives, and so obtains the gradient of every weight in a single pass. That idea, published in 1986, is what makes training deep networks possible.
Gradient descent is the algorithm that trains almost every neural network. It computes the slope of the loss function with respect to each weight and takes a small step in the opposite direction, controlled by the learning rate, until it reaches a minimum where the error stops falling and the model has converged.
L1 and L2 regularization adds a term to the loss function that penalises large weights. L2, or weight decay, shrinks the weights smoothly towards zero; L1 drives them exactly to zero and yields sparse models. Both curb overfitting and improve the generalisation ability of a neural network on unseen data.
Binary cross-entropy is the standard loss function for two-class classification. It compares the probability returned by the sigmoid with the true label, 0 or 1, and punishes confident mistakes hard. Its formula comes from maximum likelihood, and its derivative combines with the sigmoid into a very simple gradient.
Mean absolute error (MAE) averages the absolute difference between prediction and reality, so a single outlier weighs only its fair share and never blows up the loss. Huber loss combines that robustness with the smoothness of mean squared error through a delta parameter that decides where the behaviour switches over.
A loss function measures how wrong a neural network is on a single example, comparing its prediction with the correct value. The cost function averages that loss across the whole dataset. That single number is exactly what training tries to reduce, step by step, using gradient descent to adjust every weight.
Forward propagation is the process by which a neural network turns its input into a prediction, one layer at a time. Each layer multiplies the input vector by a weight matrix, adds a bias and applies an activation function, so the output of one layer feeds the next until the final result appears at the end.
The exploding gradient problem happens when the gradient norm grows without control during backpropagation, especially in deep and recurrent networks with large weights. Training destabilises and the loss turns into NaN. Gradient clipping, together with good initialisation and normalisation, is the standard fix used today.
The vanishing gradient problem appears when the error signal shrinks as it backpropagates through many layers. The sigmoid and hyperbolic tangent functions have small derivatives, and their product tends toward zero, so the early layers barely learn. ReLU, careful initialisation and normalisation solve the problem.
Choosing an activation function is simple with one base rule: use ReLU in the hidden layers, GELU or SiLU in transformers, and reserve the output for softmax in multiclass classification, sigmoid in binary problems and a linear activation in regression. This comparison gathers formulas, ranges and use cases.
The Mish function is defined as mish(x) = x·tanh(softplus(x)). It is smooth, non-monotonic and self-regularized, properties that let it outperform Swish and ReLU across many tests. The YOLOv4 object detector adopted it in its backbone as the default activation.
The Softplus function is defined as softplus(x) = ln(1 + eˣ): a smooth, always-positive approximation of ReLU whose derivative is exactly the sigmoid. It is differentiable across its whole domain, avoids ReLU's angular kink and its output never quite reaches zero.
The ELU (Exponential Linear Unit) activation function returns x for positive inputs and α(eˣ−1) for negative ones. By allowing smooth negative values, it pushes the mean of the activations toward zero, speeds up convergence compared with ReLU, and avoids dead neurons thanks to a gradient that never vanishes completely on the negative side.
The Swish function, also called SiLU, multiplies the input by its sigmoid: swish(x) = x·σ(x). It is smooth, non-monotonic and self-gating, so it often beats ReLU in deep networks. Models like LLaMA and EfficientNet use it as their default activation.
An activation function is the nonlinear operation each neuron applies to its weighted sum z to produce its output a equals f of z. Without it, stacking layers only chains linear transformations and the whole network collapses into a single one. That nonlinearity is what lets a network learn complex patterns.
