An activation function is the part that decides, inside each neuron, how the sum of the inputs becomes an output signal. Its role looks small, but it is what separates a neural network from a plain linear regression. Without it, you could stack a thousand layers and still have the power of a single one. This article gathers and organises the activation functions we have already explained separately on the site, and answers the question that connects them all. The same explanation is available in Spanish.

Key formula $a = f(z)$

Key takeaways

  • An activation function applies a transformation, almost always nonlinear, to the weighted sum z = Wx + b to produce the neuron’s output a = f(z).
  • Without nonlinearity, a many-layer network is equivalent to a single linear layer: it cannot learn curved boundaries or complex relationships.
  • Nonlinearity is the condition that makes a network with one hidden layer a universal function approximator, a result proved in 1989.
  • There is no single best function: ReLU dominates modern hidden layers, while sigmoid and softmax are reserved for the output.
  • Choosing well affects training speed and known problems such as the vanishing gradient.

What is an activation function?

Each neuron does two things. First it combines its inputs: it multiplies each input value by its weight, sums them and adds a bias, giving a number $z = Wx + b$. That number can be anything, from minus infinity to plus infinity. The activation function is the second step: it takes that $z$ and turns it into the neuron’s final output, $a = f(z)$.

The name comes from the biological analogy. In a real neuron, the input signals must cross a threshold for the cell to "fire". The earliest activation functions imitated that all-or-nothing behaviour. Modern functions are smoother, but they keep the idea: modulating how much passes to the next layer. For a general reference you can consult the Wikipedia article on the activation function[1].

Why nonlinearity is essential

Here is the heart of the matter. Suppose we remove the activation function, or use a linear one such as $f(z) = z$. Then each layer only multiplies by a matrix and adds a vector. But the composition of two linear transformations is still a linear transformation: chaining two layers reorders into

$$W_2\,(W_1 x + b_1) + b_2 = (W_2 W_1)\,x + (W_2 b_1 + b_2) = W’ x + b’$$

that is, there is a single equivalent matrix that does the same job. As a result, a 50-layer network without a nonlinear activation would have exactly the same capacity as a 1-layer network. All the effort of depth would be wasted.

See why two linear layers collapse into one

With two linear layers and no activation: $h = W_1 x + b_1$ and $y = W_2 h + b_2$. Substituting the first into the second, $y = W_2\,(W_1 x + b_1) + b_2 = (W_2 W_1)\,x + (W_2 b_1 + b_2)$. If we call $W’ = W_2 W_1$ and $b’ = W_2 b_1 + b_2$, we are left with $y = W’ x + b’$: a single equivalent linear layer, no matter how many you stacked.

Nonlinearity breaks that trap. By introducing a curve, a kink or a jump between layers, the network can bend the decision space and separate data that a straight line never could. That is the argument of the universal approximation theorem: with a nonlinear activation function, a network with a single hidden layer and enough neurons can approximate any continuous function to whatever precision you want. You can read the statement of the universal approximation theorem[2] for the details. As the reference book by Goodfellow, Bengio and Courville puts it, «in modern neural networks, the default recommendation is to use the rectified linear unit».

The universal approximation theorem guarantees that a network capable of approximating the function exists, but it does not say how to find its weights: finding them is the job of gradient-descent training, which needs a differentiable activation function.

A tour of the activation functions

These are the functions we have covered in detail, ordered from the oldest to the most common today:

  • Step: returns 0 or 1 depending on a threshold. It is intuitive but has no useful derivative, so it cannot be trained with gradients.
  • Sigmoid: squashes any value into the range between 0 and 1, ideal for reading outputs as probabilities.
  • Hyperbolic tangent: similar to the sigmoid but centred on zero, with output between -1 and 1.
  • ReLU: returns 0 if the input is negative and the value as is if it is positive. It is fast and largely avoids the vanishing gradient.
  • Leaky ReLU: a variant that lets a small slope through for negative inputs and stops the neuron from "dying".
  • Softmax: turns a vector of numbers into a probability distribution that sums to 1, the standard choice in a classifier’s output layer.

How to choose

The choice depends on where the neuron is. For hidden layers, the default recommendation for more than a decade has been ReLU: it is cheap to compute and trains fast. It was one of the keys to deep learning’s performance jump around 2012. If you see neurons that stop learning, Leaky ReLU or its variants usually fix it.

For the output layer, the problem sets the function. A binary classification uses a sigmoid, giving a number between 0 and 1 you can read as a probability. A multi-class classification uses softmax, spreading probability across all categories. A regression, where the output is any real number, often applies no activation at all. Sigmoid and tanh have nearly vanished from deep hidden layers because they saturate and slow the gradient, a problem documented in texts such as Neural Networks and Deep Learning[3].

Visual examples

Imagine a neuron receiving $z = 2$. The sigmoid turns it into roughly $0.88$; the tanh, into about $0.96$; and ReLU leaves it at $2$. Now imagine $z = -3$. The sigmoid returns almost $0.05$; the tanh, close to $-0.99$; and ReLU cuts it flat to $0$. That different behaviour for the same value is what gives each function its character.

The shape of the curve matters. The sigmoid and the tanh are smooth and bounded, S-shaped, which is why they saturate at the extremes. ReLU is a broken line: flat to the left of zero and with slope 1 to the right. That simple geometry explains why its derivative is so cheap (it is $0$ or $1$) and why it dominates today’s deep networks.

Frequently asked questions

What happens if I use no activation function?

The network behaves as a linear model, no matter how many layers it has. You could fit a line or a plane, but not a curved boundary. In practice, a deep network without nonlinear activations adds nothing over a simple linear regression.

Which is the best activation function?

There is no single best one. For hidden layers, ReLU is the recommended starting point for its speed. For the output, the problem picks it: sigmoid for binary classification and softmax for several classes. The correct answer is almost always "it depends on the layer and the task".

Why is the sigmoid no longer used in hidden layers?

Because it saturates: when the input is very large or very small, its derivative approaches zero and the gradient nearly vanishes as it backpropagates through many layers. This slows or halts learning in deep networks, something ReLU mitigates by keeping a constant slope of 1 in the positive region.

Conclusion

The activation function is the part that turns a stack of matrix multiplications into a machine able to learn almost anything. Its secret is not the specific formula but the nonlinearity it introduces between layers. With that principle clear, each particular function (step, sigmoid, tanh, ReLU, Leaky ReLU or softmax) stops being a loose name and finds its place. The natural next step is to review each one in detail and see how it fits into the mathematics behind neural networks.

Sources

  1. Wikipedia article on the activation function
  2. universal approximation theorem
  3. Neural Networks and Deep Learning
  4. Deep Learning (Goodfellow, Bengio and Courville)

Route: The Neuron and Activation Functions