The SELU Activation and Self-Normalizing Networks

The SELU (Scaled Exponential Linear Unit) function is the activation that lets a deep network normalize itself, with no need for batch normalization. It is an ELU multiplied by a scale factor greater than 1, and that seemingly small detail has a huge consequence: if you pick the constants right, activations keep zero mean and unit variance as they pass through the layers. That property is called self-normalization. The same explanation is available in Spanish.

Key takeaways

• The SELU activation function is defined as SELU(x) = λ · ELU(x), that is, an ELU scaled by a factor λ greater than 1.
• Its two constants have fixed, very precise values: λ ≈ 1.0507 and α ≈ 1.6733, chosen so activations converge towards zero mean and unit variance.
• It was introduced by Klambauer, Unterthiner, Mayr and Hochreiter in 2017, with an appendix of more than 90 pages of mathematical proofs.
• For self-normalization to work you need LeCun initialization (variance 1/n) and, if you use dropout, the alpha-dropout variant.
• Its negative saturation value is −λα ≈ −1.7581, a lower bound that stabilizes the signal variance.

What is SELU?

SELU is an activation function, the non-linear operation applied to each neuron’s z = Wx + b output before it is passed to the next layer. Without that non-linearity, stacking layers would achieve nothing: a stack of linear transformations is still linear. To place this piece within the whole picture, the mathematics behind neural networks roadmap shows where each function fits.

What sets SELU apart is not its shape, which is almost identical to an ELU, but its purpose. Most activations simply introduce non-linearity. SELU goes one step further: it is designed to control the mean and variance of activations layer after layer. That control is what gives Self-Normalizing Networks (SNNs) their name.

Formula and the constants lambda and alpha

The definition reuses the ELU and adds a scale factor:

$$\operatorname{SELU}(x) = \lambda \begin{cases} x & \text{if } x > 0 \ \alpha\,(e^x-1) & \text{if } x \le 0 \end{cases}$$

The part in parentheses is exactly an ELU with parameter $\alpha$. The novelty is the factor $\lambda$ that multiplies everything. Its values are not free: they come from solving the fixed-point equations that guarantee self-normalization, and they are $\lambda \approx 1.0507009873554805$ and $\alpha \approx 1.6732632423543772$.

With concrete numbers: $\operatorname{SELU}(1) \approx 1.0507$, $\operatorname{SELU}(0) = 0$ and $\operatorname{SELU}(-2) \approx -1.5202$. Notice two things. For positive inputs, the slope is $\lambda \approx 1.0507$, slightly greater than 1, so the signal is amplified a little. For very negative inputs, the output tends to $-\lambda\alpha \approx -1.7581$, a floor that keeps the variance from blowing up. That balance between amplifying and saturating is the heart of the mechanism.

SELU inherits the ELU’s derivative, scaled by $\lambda$: it equals $\lambda$ in the positive region and $\lambda\alpha\,e^x$ in the negative one, so it never vanishes and the gradient always flows.

Self-normalization

The idea that makes SELU special is the fixed point. Klambauer and his co-authors prove that, if a layer’s input has mean 0 and variance 1, the output after applying SELU also tends to mean 0 and variance 1. In other words, (0, 1) is a stable fixed point of the map that transforms mean and variance from one layer to the next. As the authors write, «activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance, even under the presence of noise and perturbations».

This replaces batch normalization. In a normal network, activations tend to explode or vanish as layers accumulate, and to prevent that we insert normalization layers that recenter the signal at each step. SELU achieves the same effect by construction: the normalization is baked into the function itself. That is why it can train dense networks of dozens of layers that, with classic activations, would be unstable.

Requirements: LeCun initialization and alpha-dropout

Self-normalization is not magic: it only holds if you respect two conditions. This is where many experiments fail.

The first is LeCun normal initialization: weights must start from a normal distribution with mean 0 and variance 1/n, where n is the number of inputs to the neuron. That specific variance is what makes the signal enter each layer with the statistics the fixed point expects. If you use He or Glorot initialization, the guarantee breaks.

The second concerns dropout. Classic dropout sets a fraction of activations to 0, which shifts the mean and variance and ruins self-normalization. The fix is alpha-dropout, which instead of zeroing the dropped units pushes them towards the saturation value −λα ≈ −1.7581 and rescales the rest, so mean and variance are preserved. In PyTorch you get both pieces ready to use.

import torch
import torch.nn as nn

selu = nn.SELU()
x = torch.tensor([-2.0, 0.0, 1.0])
print(selu(x))              # tensor([-1.5202,  0.0000,  1.0507])

block = nn.Sequential(    # self-normalizing Linear + SELU + AlphaDropout
    nn.Linear(128, 128),
    nn.SELU(),
    nn.AlphaDropout(p=0.1),
)

The output matches the values we computed by hand, a good way to confirm you have understood the formula.

SELU vs ELU

SELU and ELU share the same curve, but not the same goal. The difference comes down to the factor λ:

• ELU (x if x > 0, α(eˣ − 1) if x ≤ 0) has slope 1 in the positive region and saturates at −α. Its aim is to avoid ReLU’s dead neurons by letting some negative signal through.
• SELU is that same ELU multiplied by λ ≈ 1.0507, with α fixed at ≈ 1.6733. Its added aim is to keep the variance stable, not just to smooth the response.

The positive slope greater than 1 is not a whim: it amplifies activations slightly to offset the contraction the negative part introduces, so the total variance neither grows nor shrinks. In ELU those constants are free; in SELU they are computed precisely to close the self-normalization loop.

Frequently asked questions

How does SELU differ from ELU?

SELU is an ELU multiplied by a scale factor λ ≈ 1.0507, with the parameter α fixed at ≈ 1.6733. That scaling makes the positive slope greater than 1 and keeps activations at zero mean and unit variance across layers, something plain ELU does not guarantee.

Why does SELU need LeCun initialization?

Because the self-normalization proof assumes the signal reaches each layer with variance 1, and that only happens if weights start from a normal with variance 1/n. LeCun initialization provides exactly that starting condition; with He or Glorot, the fixed point (0, 1) no longer holds.

Does SELU replace batch normalization?

That is its promise: by normalizing through construction, a SELU network can drop the batch normalization layers. In deep dense networks it works very well, but in convolutional architectures or modern transformers other techniques usually perform better, so SELU has not fully displaced explicit normalization.

Conclusion

SELU turns an ELU into a free normalization mechanism: just multiply it by λ ≈ 1.0507 and fix α ≈ 1.6733 so activations stay at zero mean and unit variance layer after layer. Understanding its formula SELU(x) = λ · ELU(x) and its two requirements, LeCun initialization and alpha-dropout, lets you build deep, stable dense networks. The natural next step is to compare this idea with the GELU function, the activation that came to dominate transformers.

Sources

1. Self-Normalizing Neural Networks, Klambauer et al. (arXiv, 2017)
2. Rectifier (neural networks) (Wikipedia)
3. torch.nn.SELU (PyTorch documentation)
4. tf.keras.activations.selu (TensorFlow documentation)

Route: The Neuron and Activation Functions