Tag
The vanishing gradient problem appears when the error signal shrinks as it backpropagates through many layers. The sigmoid and hyperbolic tangent functions have small derivatives, and their product tends toward zero, so the early layers barely learn. ReLU, careful initialisation and normalisation solve the problem.
Choosing an activation function is simple with one base rule: use ReLU in the hidden layers, GELU or SiLU in transformers, and reserve the output for softmax in multiclass classification, sigmoid in binary problems and a linear activation in regression. This comparison gathers formulas, ranges and use cases.
The Mish function is defined as mish(x) = x·tanh(softplus(x)). It is smooth, non-monotonic and self-regularized, properties that let it outperform Swish and ReLU across many tests. The YOLOv4 object detector adopted it in its backbone as the default activation.
The Softplus function is defined as softplus(x) = ln(1 + eˣ): a smooth, always-positive approximation of ReLU whose derivative is exactly the sigmoid. It is differentiable across its whole domain, avoids ReLU's angular kink and its output never quite reaches zero.
The SELU (Scaled Exponential Linear Unit) function is defined as SELU(x) equal to lambda times ELU(x), with lambda near 1.0507 and alpha near 1.6733. Those two constants make activations converge on their own towards zero mean and unit variance layer after layer, building deep networks that normalize themselves without batch normalization.
The ELU (Exponential Linear Unit) activation function returns x for positive inputs and α(eˣ−1) for negative ones. By allowing smooth negative values, it pushes the mean of the activations toward zero, speeds up convergence compared with ReLU, and avoids dead neurons thanks to a gradient that never vanishes completely on the negative side.
The Swish function, also called SiLU, multiplies the input by its sigmoid: swish(x) = x·σ(x). It is smooth, non-monotonic and self-gating, so it often beats ReLU in deep networks. Models like LLaMA and EfficientNet use it as their default activation.
The GELU (Gaussian Error Linear Unit) function multiplies each input by the probability that a standard normal falls below that input. The result is a smooth curve with a continuous derivative that weights inputs by their magnitude, and it has become the default activation inside BERT and GPT.
