Tag
Weight initialization sets the starting scale of the matrix W before training begins. Xavier/Glorot, from 2010, splits the variance between inputs and outputs and suits sigmoid and tanh; He, from 2015, doubles it for ReLU, which zeroes half the activations. A poor choice stalls or breaks learning.
Choosing an activation function is simple with one base rule: use ReLU in the hidden layers, GELU or SiLU in transformers, and reserve the output for softmax in multiclass classification, sigmoid in binary problems and a linear activation in regression. This comparison gathers formulas, ranges and use cases.
Leaky ReLU is a variant of the ReLU function that replaces zero for negative values with a small slope, keeping neurons from ever fully shutting down. This solves the dying neuron problem and improves training stability in deep neural networks, CNNs, and GAN discriminators.
ReLU (rectified linear unit) is the most widely used activation function in deep neural networks: it returns the input value if positive, and zero if negative, defined as f(x) = max(0, x). Its low computational cost and resistance to the vanishing gradient problem that hampers sigmoid made it the de facto standard since AlexNet in 2012.
