The ELU (Exponential Linear Unit) activation behaves like the identity for positive inputs and decays smoothly toward a negative value when the input is less than or equal to zero. It was proposed in 2015 by Djork-Arné Clevert, Thomas Unterthiner and Sepp Hochreiter, and it aims to combine the best of ReLU (a stable gradient on the positive side) with the benefit of negative outputs, which push the mean of the activations toward zero and speed up learning. The same explanation is available in Spanish.

Key formula $\operatorname{ELU}(x) = \begin{cases} x & x > 0 \\ \alpha(e^x-1) & x \le 0 \end{cases}$

Key takeaways

  • The ELU activation function returns z if z > 0 and α(eˣ−1) if z ≤ 0, where z = Wx + b is the neuron’s weighted sum.
  • For negative values it saturates smoothly toward −α, which lowers noise and keeps the mean of the activations near zero.
  • Its derivative is 1 on the positive side and α·eˣ on the negative one, so the gradient never reaches 0 and dead neurons do not appear as they do with ReLU.
  • The hyperparameter α controls the saturation; the default value is 1, both in the original paper and in PyTorch.
  • In the original paper, an ELU network reached 24.28% test error on CIFAR-100, the best published result at the time.

What is the ELU function

An activation function decides how a neuron transforms its input before passing it to the next layer. Without one, stacking layers would collapse into a single linear transformation unable to model complex relationships. If this is new to you, it helps to review what an activation function is first.

ELU exists to fix two flaws of the ReLU function. ReLU sets any negative input to zero, which causes two problems: the mean of the outputs is pushed toward positive values, and when a neuron always receives negative inputs its gradient becomes zero and it stops learning (the "dead neuron" phenomenon). ELU replaces that flat part with an exponential curve that does let negative information through.

Formula and derivative

ELU is defined piecewise over the pre-activation $z$:

$$\operatorname{ELU}(z) = \begin{cases} z & z > 0 \ \alpha(e^z-1) & z \le 0 \end{cases}$$

Here $e^z$ is the natural exponential evaluated at $z$ and $\alpha > 0$. With $\alpha = 1$, as $z$ tends to minus infinity the $e^z$ term tends to $0$, so the output saturates toward $-1$. That controlled floor is exactly what gives robustness against noise: for instance, $\operatorname{ELU}(-2) \approx -0.8647$, $\operatorname{ELU}(0) = 0$ and $\operatorname{ELU}(2) = 2$.

The derivative, which we need for backpropagation, is also simple:

$$\operatorname{ELU}'(z) = \begin{cases} 1 & z > 0 \ \alpha e^z = \operatorname{ELU}(z) + \alpha & z \le 0 \end{cases}$$

Unlike ReLU, the ELU derivative is never exactly $0$ on the negative branch: it only tends to $0$ as $z \to -\infty$. That is why an ELU neuron always keeps some gradient and can recover during training.

On the positive side the gradient is exactly $1$, just like ReLU, so there is no vanishing gradient in the active region. On the negative side the gradient is small but nonzero, which lets the network recover neurons that ReLU would have left inactive forever.

See why the derivative is ELU itself plus α

On the negative branch, $\operatorname{ELU}(z) = \alpha(e^z-1)$. Differentiating with respect to $z$, the constant $-\alpha$ vanishes and we are left with $\dfrac{d}{dz}\,\alpha(e^z-1) = \alpha e^z$. And since $\alpha e^z = \alpha(e^z-1) + \alpha = \operatorname{ELU}(z) + \alpha$, the derivative can be written from the output value itself, a trick that avoids re-evaluating the exponential.

Mean near zero and convergence

The paper’s central argument is that pushing the mean of the activations toward zero speeds up training, because it reduces the so-called bias shift between layers. It is the same goal as batch normalization, but ELU achieves it through the shape of the function itself, without extra per-minibatch statistics. As the authors summarise:

«ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity» (Clevert, Unterthiner and Hochreiter, 2015).

In their experiments, ELU networks converged faster and generalised better than the equivalent ReLU ones. On CIFAR-100 they reached that 24.28% error, and on ImageNet a deep convolutional network with ELU outperformed its ReLU version without needing batch normalization. The work was presented at the ICLR 2016 conference.

ELU versus ReLU and Leaky ReLU

The three functions agree on the positive side: they return the input unchanged. The difference is how they treat negative values.

  • ReLU zeroes them out entirely. It is the fastest to compute, but it suffers from dead neurons and a shifted mean.
  • Leaky ReLU lets a small linear slope through (for example 0.01·z) for negatives, which avoids the zero gradient but keeps an abrupt response at the origin.
  • ELU uses an exponential curve, so it is continuous and differentiable across its whole domain and saturates toward −α, making it more robust to very negative inputs.

The price of ELU is computing the exponential, which is costlier than a comparison or a multiplication. In practice that difference is usually acceptable, and many architectures choose it when training stability matters more than a few milliseconds per step.

The alpha hyperparameter

The α parameter sets the value the negative branch saturates toward. With α = 1, the most common choice, the output never drops below −1. If you increase α, you allow larger negative outputs and a later saturation; if you lower it, the function looks more like ReLU. In most frameworks, as the PyTorch[1] documentation shows, α is a fixed value that defaults to 1 and is not learned during training. A later variant, PELU, does treat α as a trainable parameter.

In PyTorch, instantiating the function is as direct as this:

import torch.nn as nn

activation = nn.ELU(alpha=1.0)

Frequently asked questions

When should I use ELU instead of ReLU?

When training is unstable or many dead neurons appear, ELU tends to help, because its negative branch keeps a nonzero gradient and centres the activations. If the cost of computing the exponential is a problem at inference time, ReLU remains the default choice for its simplicity.

What value of alpha should I pick?

Always start with α = 1, the value recommended in the original paper and the default in the frameworks. It is only worth tuning if you have experimental evidence that another saturation improves your results; changing it without measuring rarely pays off.

Does ELU solve the vanishing gradient problem?

It mitigates it on the negative side, where its derivative α·eˣ is small but never zero, unlike ReLU. On the positive side the gradient is 1, so it does not vanish there either. It is not a full guarantee, but it lowers the risk compared with activations like the sigmoid.

Conclusion

The ELU function is an activation designed to train deep networks more stably: it keeps ReLU’s unit gradient on the positive side and adds an exponential branch that centres the activations and removes dead neurons. Its only real cost is computing the exponential. To place it within the wider mathematical toolkit, the best starting point is the mathematics of neural networks roadmap, and from there you can move on to the other activation functions.

Sources

  1. PyTorch
  2. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), Clevert et al., 2015
  3. Activation function (Wikipedia)

Route: The Neuron and Activation Functions