Archive
Categorical cross-entropy is the standard loss function for classifying into several mutually exclusive classes. It compares the distribution the network predicts, usually after a softmax, with the true label written in one-hot format, and it penalises heavily any low probability assigned to the correct class.
Binary cross-entropy is the standard loss function for two-class classification. It compares the probability returned by the sigmoid with the true label, 0 or 1, and punishes confident mistakes hard. Its formula comes from maximum likelihood, and its derivative combines with the sigmoid into a very simple gradient.
Mean absolute error (MAE) averages the absolute difference between prediction and reality, so a single outlier weighs only its fair share and never blows up the loss. Huber loss combines that robustness with the smoothness of mean squared error through a delta parameter that decides where the behaviour switches over.
Mean squared error (MSE) is the most common loss function in regression: it averages the square of the gap between each true value and its prediction. Squaring punishes big mistakes hard, keeps the function differentiable and gives gradient descent a clear, smooth target to minimise while training a model.
A loss function measures how wrong a neural network is on a single example, comparing its prediction with the correct value. The cost function averages that loss across the whole dataset. That single number is exactly what training tries to reduce, step by step, using gradient descent to adjust every weight.
Forward propagation is the process by which a neural network turns its input into a prediction, one layer at a time. Each layer multiplies the input vector by a weight matrix, adds a bias and applies an activation function, so the output of one layer feeds the next until the final result appears at the end.
The exploding gradient problem happens when the gradient norm grows without control during backpropagation, especially in deep and recurrent networks with large weights. Training destabilises and the loss turns into NaN. Gradient clipping, together with good initialisation and normalisation, is the standard fix used today.
The vanishing gradient problem appears when the error signal shrinks as it backpropagates through many layers. The sigmoid and hyperbolic tangent functions have small derivatives, and their product tends toward zero, so the early layers barely learn. ReLU, careful initialisation and normalisation solve the problem.
Choosing an activation function is simple with one base rule: use ReLU in the hidden layers, GELU or SiLU in transformers, and reserve the output for softmax in multiclass classification, sigmoid in binary problems and a linear activation in regression. This comparison gathers formulas, ranges and use cases.
The Mish function is defined as mish(x) = x·tanh(softplus(x)). It is smooth, non-monotonic and self-regularized, properties that let it outperform Swish and ReLU across many tests. The YOLOv4 object detector adopted it in its backbone as the default activation.
The Softplus function is defined as softplus(x) = ln(1 + eˣ): a smooth, always-positive approximation of ReLU whose derivative is exactly the sigmoid. It is differentiable across its whole domain, avoids ReLU's angular kink and its output never quite reaches zero.
The SELU (Scaled Exponential Linear Unit) function is defined as SELU(x) equal to lambda times ELU(x), with lambda near 1.0507 and alpha near 1.6733. Those two constants make activations converge on their own towards zero mean and unit variance layer after layer, building deep networks that normalize themselves without batch normalization.
The ELU (Exponential Linear Unit) activation function returns x for positive inputs and α(eˣ−1) for negative ones. By allowing smooth negative values, it pushes the mean of the activations toward zero, speeds up convergence compared with ReLU, and avoids dead neurons thanks to a gradient that never vanishes completely on the negative side.
The Swish function, also called SiLU, multiplies the input by its sigmoid: swish(x) = x·σ(x). It is smooth, non-monotonic and self-gating, so it often beats ReLU in deep networks. Models like LLaMA and EfficientNet use it as their default activation.
The GELU (Gaussian Error Linear Unit) function multiplies each input by the probability that a standard normal falls below that input. The result is a smooth curve with a continuous derivative that weights inputs by their magnitude, and it has become the default activation inside BERT and GPT.
An activation function is the nonlinear operation each neuron applies to its weighted sum z to produce its output a equals f of z. Without it, stacking layers only chains linear transformations and the whole network collapses into a single one. That nonlinearity is what lets a network learn complex patterns.
In an artificial neuron, weights measure how important each input is and the bias shifts the result. The neuron multiplies each input by its weight, adds everything up and includes the bias to produce the weighted sum z = Wx + b, the number that then passes through the activation function.
The perceptron is the simplest artificial neuron: it takes several inputs, multiplies them by its weights, adds a bias and applies an activation function that decides between two outputs. Frank Rosenblatt introduced it in 1958, and it remains the basic building block of every modern neural network.
Probability in neural networks rests on three ideas: a distribution assigns weights to possible outcomes, the expected value averages those outcomes, and likelihood measures how well the model fits the data. From those three pieces come the softmax function and the cross-entropy loss that we optimise.
The exponential function eˣ and its inverse, the natural logarithm ln(x), appear again and again in deep learning. The exponential builds the sigmoid and softmax that turn numbers into probabilities, while the natural logarithm defines cross-entropy, the loss used to train almost every classifier in practice today.
A partial derivative measures how a function changes when you move just one of its variables and hold the rest fixed. The gradient gathers all those partial derivatives into a vector that points toward the steepest ascent; in a neural network, moving the opposite way lowers the error and drives the training.
The chain rule computes the derivative of a composite function by multiplying the derivatives of its links: if y depends on u and u depends on x, then dy/dx equals dy/du times du/dx. That layer-by-layer multiplication of derivatives is exactly what backpropagation does to train a neural network.
The essential differentiation rules are a handful of formulas that turn any function into its derivative: the power rule, the product and quotient rules, and the rules for the exponential and the logarithm. With them, plus the chain rule, a neural network computes gradients and learns by adjusting its weights.
A derivative measures the rate of change of a function: how much its output varies when the input changes a little. In a neural network, that slope tells us in which direction and how strongly to adjust each weight to reduce the error, and it is the foundation of gradient descent and backpropagation.
