How to Choose an Activation Function (Comparison)
Table of contents
- Key takeaways
- Comparison table of activation functions
- Hidden layers: ReLU and its family
- Output layer: sigmoid, softmax, linear
- Transformers and LLMs: GELU and SiLU
- Practical rules for choosing
- Frequently asked questions
- Which activation function should I use by default?
- When should I use GELU or SiLU instead of ReLU?
- Which activation goes in the output layer?
- Conclusion
- Sources
There is no single perfect activation function: there is the right one for each layer. Getting that choice right speeds up training and avoids classic problems such as the vanishing gradient or dead neurons. This comparison gathers the most used functions, with their formulas and ranges, and offers a practical rule for each case. If you prefer Spanish, here is the Spanish version. For the full context, see the mathematics of neural networks roadmap and the article on what an activation function is.
Key takeaways
- For hidden layers, ReLU is the default option since 2010: it is cheap to compute and works well in most networks.
- In transformers and large language models, GELU and SiLU are preferred, smooth versions of ReLU that tend to improve convergence.
- The output layer depends on the task: sigmoid for binary classification, softmax for multiclass and a linear activation for regression.
- Problems with dead neurons or a vanishing gradient are almost always fixed by changing the activation or picking a variant.
- Always start with ReLU and change it only if you measure a real gain: the team that proposed Swish measured just a 0.9% improvement over ReLU on ImageNet.
Comparison table of activation functions
This table sums up the eight most common activations, with their formula, the range of values they return and their main use. All of them operate on the weighted sum $\mathbf{z} = \mathbf{W}\mathbf{x}+\mathbf{b}$ and produce the activation $a = f(z)$.
| Function | Formula | Range | Main use |
|---|---|---|---|
| Sigmoid | $\dfrac{1}{1+e^{-z}}$ | $(0, 1)$ | Binary output and LSTM gates |
| Tanh | $\dfrac{e^z-e^{-z}}{e^z+e^{-z}}$ | $(-1, 1)$ | Classic hidden layers and recurrent networks |
| ReLU | $\max(0, z)$ | $[0, \infty)$ | Default hidden layers (CNN, MLP) |
| Leaky ReLU | $\max(\alpha z, z)$ | $(-\infty, \infty)$ | Avoiding dead neurons |
| GELU | $z \cdot \Phi(z)$ | $(\approx-0.17, \infty)$ | Transformers such as BERT and GPT |
| SiLU/Swish | $z \cdot \sigma(z)$ | $(\approx-0.28, \infty)$ | Deep and vision networks (EfficientNet) |
| Softmax | $\dfrac{e^{z_i}}{\sum_j e^{z_j}}$ | $(0, 1)$, sums to 1 | Multiclass output layer |
| Linear | $z$ | $(-\infty, \infty)$ | Output layer in regression |
Base rule: always start with ReLU in the hidden layers and change activation only when you measure a real gain on validation, not by fashion.
Hidden layers: ReLU and its family
In hidden layers, the default choice is the ReLU function. Its definition, $\max(0, z)$, is so cheap to compute that it has dominated since the work of Nair and Hinton in 2010, and it avoids the saturation that hurt sigmoid and tanh in deep networks.
Its only weak point is dead neurons: a unit whose input is always negative stops learning because its gradient is $0$. When that happens, it is worth switching to the Leaky ReLU function, which lets a small slope $\alpha$ (for example $0.01$) through for negative inputs. Other variants of the same family are ELU and SELU, useful when you want more stable normalisation.
Output layer: sigmoid, softmax, linear
The activation of the last layer is not chosen by taste but by the shape of the answer. For binary classification (spam or not spam) you use the sigmoid function, which squeezes any number into the interval (0, 1) and reads as a probability.
To classify among several mutually exclusive categories, the output is the softmax function: it turns a vector of numbers into a distribution that sums to exactly 1. And if the target is a continuous value, such as a price or a temperature, the last layer carries no activation: a linear output lets the number pass through unchanged.
Transformers and LLMs: GELU and SiLU
Modern architectures smooth the ReLU corner. The GELU function, proposed in 2016, multiplies the input by the cumulative probability of a normal, $z \cdot \Phi(z)$, and is the standard activation of BERT and the GPT family. The SiLU function, also called Swish, follows the same idea with $z \cdot \sigma(z)$ and appears in vision networks such as EfficientNet. Compared with ReLU, they provide a continuous gradient that helps train very deep networks.
The gain is real but moderate. As the Google Brain team that presented Swish in 2017 concluded, «simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9%». That is why the practical recommendation is to reserve GELU and SiLU for transformers and large networks, where that margin pays off the extra compute cost.
Practical rules for choosing
- Start with ReLU in every hidden layer.
- If dead neurons appear, switch to Leaky ReLU or ELU.
- In a transformer or a language model, use GELU or SiLU.
- In the output, choose by task: sigmoid (binary), softmax (multiclass) or linear (regression).
- Change an activation only when you measure a gain on validation, not by fashion.
Frequently asked questions
Which activation function should I use by default?
ReLU in the hidden layers. It is fast, stable and the recommended starting point in almost any convolutional network or multilayer perceptron. You should only change it if you detect dead neurons or if you work with a transformer.
When should I use GELU or SiLU instead of ReLU?
In very deep networks and, above all, in transformers and large language models. GELU is the standard activation of BERT and GPT, and SiLU shines in vision networks. Their smooth gradient improves convergence, although the measured gain usually stays below 1%.
Which activation goes in the output layer?
It depends on the task: sigmoid for binary classification, softmax for multiclass classification and no activation (linear output) for regression. The rule is that the shape of the output must match the shape of the answer you expect.
Conclusion
Choosing an activation function comes down to two questions: which layer you are defining and which task the network solves. For hidden layers, ReLU is the starting point and GELU or SiLU the next step in large models; for the output, the task decides. With that comparison in mind, the decision stops being intuition and becomes method. A good next step is to review the softmax function and its role in classification.
