Tag
Weight initialization sets the starting scale of the matrix W before training begins. Xavier/Glorot, from 2010, splits the variance between inputs and outputs and suits sigmoid and tanh; He, from 2015, doubles it for ReLU, which zeroes half the activations. A poor choice stalls or breaks learning.
Choosing an activation function is simple with one base rule: use ReLU in the hidden layers, GELU or SiLU in transformers, and reserve the output for softmax in multiclass classification, sigmoid in binary problems and a linear activation in regression. This comparison gathers formulas, ranges and use cases.
Leaky ReLU is an activation function derived from ReLU that replaces the zero output for negative inputs with a small slope, typically 0.01. This keeps the gradient from ever reaching zero, which prevents the dying neuron problem and stabilizes training in very deep convolutional, recurrent, and GAN networks.
ReLU (Rectified Linear Unit, f(x) = max(0, x)) has been the dominant activation function in deep learning since AlexNet popularised it in 2012: cheap to compute, resistant to vanishing gradients, and with one well-known weakness, dying ReLU.
