Tag
The hyperbolic tangent (tanh) is an activation function that squashes any real value into the interval (-1, 1) with zero-centred output, which removes the systematic gradient bias seen with sigmoid. It is the standard activation inside the LSTM and GRU memory cells used in recurrent networks.
The sigmoid function maps any real number to a value between 0 and 1, which makes it the natural activation function for expressing probabilities in a neural network. It is differentiable across its whole domain, though it suffers from saturation and vanishing gradients in deep networks, so today it is mostly reserved for the output layer.
Leaky ReLU is an activation function derived from ReLU that replaces the zero output for negative inputs with a small slope, typically 0.01. This keeps the gradient from ever reaching zero, which prevents the dying neuron problem and stabilizes training in very deep convolutional, recurrent, and GAN networks.
ReLU (Rectified Linear Unit, f(x) = max(0, x)) has been the dominant activation function in deep learning since AlexNet popularised it in 2012: cheap to compute, resistant to vanishing gradients, and with one well-known weakness, dying ReLU.
The step function, or Heaviside function, is the simplest activation function in a neural network: it maps any input to a binary output, 0 or 1, depending on whether it crosses a fixed threshold. It was the core decision mechanism of Rosenblatt's perceptron in 1958, but its derivative is zero almost everywhere, so modern networks use sigmoid or ReLU instead.
