The GELU (Gaussian Error Linear Unit) function is the default activation inside BERT, GPT and most modern transformers. Instead of cutting negative values off sharply the way ReLU does, GELU dampens them gradually: it multiplies each number by the probability that a standard normal variable falls below it. The result is a smooth curve that improves training in very deep models. The same explanation is available in Spanish.

Key formula $\mathrm{GELU}(x) = x \cdot \Phi(x)$

Key takeaways

  • The GELU activation function is defined as GELU(x) = x · Φ(x), where Φ(x) is the cumulative distribution function of the standard normal.
  • Unlike ReLU, GELU is smooth and has a continuous derivative across its whole domain, which helps make gradient descent more stable.
  • It was proposed by Dan Hendrycks and Kevin Gimpel in 2016 and is the default activation in BERT (2018) and the GPT family.
  • It weights each input by its magnitude instead of deciding purely by sign, so small negative values keep part of their information.
  • In practice it is computed with a hyperbolic-tangent approximation that avoids evaluating the Gaussian integral at every step.

What is GELU?

GELU is an activation function, the non-linear operation applied to each neuron’s z = Wx + b output before it is passed to the next layer. Without that non-linearity, stacking layers would achieve nothing: a stack of linear transformations is still linear. To place this piece within the whole picture, the mathematics behind neural networks roadmap shows where each function fits.

The idea behind GELU is to treat the activation as a probabilistic decision. Instead of multiplying the input by 0 or 1 based on its sign (what ReLU does), it multiplies by a number between 0 and 1: the probability Φ(x) that a sample from the standard normal is smaller than x. The larger x is, the closer that probability is to 1 and the more the input passes through; the more negative it is, the closer to 0 and the more it is dampened. As Hendrycks and Gimpel write, «the GELU nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLUs».

Formula and derivative

The exact definition uses the cumulative distribution function $\Phi$ of the standard normal:

$$\mathrm{GELU}(x) = x \cdot \Phi(x) = x \cdot \tfrac{1}{2}\left[1 + \operatorname{erf}!\left(\tfrac{x}{\sqrt{2}}\right)\right]$$

Here $\operatorname{erf}$ is the error function, which is where the word "Error" in the name comes from. To see the behaviour with concrete numbers: $\mathrm{GELU}(-2) \approx -0.0455$, $\mathrm{GELU}(0) = 0$ and $\mathrm{GELU}(2) \approx 1.9545$. Notice that the output at $x = -2$ is not zero, as it would be with ReLU, but a small negative value.

GELU’s derivative is continuous across its whole domain, including at the origin. That removes the “kink” ReLU has at zero, where its derivative jumps abruptly from 0 to 1.

Because the curve is smooth, its derivative exists everywhere, including at the origin. That derivative is $\Phi(x) + x \cdot \varphi(x)$, where $\varphi(x)$ is the standard normal density.

Show the derivation of the derivative

Applying the product rule to $x \cdot \Phi(x)$: the first term differentiates $x$ and keeps $\Phi(x)$; the second keeps $x$ and differentiates $\Phi(x)$, whose derivative is the density $\varphi(x)$. Result: $\dfrac{d}{dx}\,\mathrm{GELU}(x) = \Phi(x) + x\,\varphi(x)$.

Having the gradient change continuously avoids the “kink” that ReLU has at zero, where the derivative jumps from 0 to 1.

## Why transformers use it (BERT, GPT)

When the Google team released BERT in 2018 it chose GELU over ReLU, and from then on nearly the whole family of large language models adopted it, including the GPT models. The practical reason is empirical: in the authors’ experiments, GELU matched or beat ReLU and ELU across vision, language and speech tasks.

Two things explain it. First, the smooth gradient helps train networks with many layers, and a large transformer stacks dozens of blocks; recall that a model like GPT-3 tunes 175 billion parameters. Second, by keeping some signal in small negative values, GELU does not “kill” neurons as bluntly as ReLU, which can leave units stuck permanently at zero. That difference shows up especially in the early stages of training.

## GELU vs ReLU and Swish

The three functions look alike for large values of `x`, where they all let the input through almost untouched. The differences sit near zero:

– **ReLU** (`max(0, x)`) is the fastest to compute, but its derivative is discontinuous at zero and it wipes out negatives entirely.
– **GELU** (`x · Φ(x)`) is smooth and lets a fraction of the negatives through according to their magnitude, with a small dip where the output goes below zero.
– **Swish** or SiLU (`x · sigmoid(x)`) has an almost identical shape to GELU; in fact, `x · sigmoid(1.702·x)` is a classic approximation of GELU.

In large language models, GELU and Swish tend to perform very similarly, and the choice depends more on the ecosystem and habit than on a clear advantage of one over the other.

## Implementation

In practice you do not evaluate the Gaussian integral at every step, but a hyperbolic-tangent approximation the authors themselves introduced:

`GELU(x) ≈ 0.5 · x · [1 + tanh(√(2/π) · (x + 0.044715 · x³))]`

That `0.044715` constant tunes the curve so it hugs the exact version. In a modern framework you do not need to write it by hand: PyTorch offers it as a ready-made layer.

import torch
import torch.nn as nn

gelu = nn.GELU()                 # exact by default; approximate='tanh' for the variant
x = torch.tensor([-2.0, 0.0, 2.0])
print(gelu(x))                   # tensor([-0.0455,  0.0000,  1.9545])

The output matches the values we computed by hand, a good way to confirm you have understood the formula.

## Frequently asked questions

### How does GELU differ from ReLU?

ReLU zeroes out any negative input and passes positives through unchanged, with an abrupt change at the origin. GELU, by contrast, is smooth: it multiplies each value by the probability `Φ(x)`, so small negatives keep part of their signal and the derivative is continuous everywhere.

### Why is GELU called “Gaussian Error Linear Unit”?

Because it combines three ideas: it is “Gaussian” in using the normal distribution, the “error” function `erf` appears in its exact definition, and it behaves like an almost “linear” unit for large values of `x`. The name summarises its formula `x · Φ(x)`.

### Is GELU slower than ReLU?

Yes, somewhat, because computing `Φ(x)` or its tanh approximation costs more than a simple `max(0, x)`. In large models that difference is small next to the total cost, and it usually pays off through the training stability it brings.

## Conclusion

GELU replaces ReLU’s abrupt decision with a smooth transition governed by the normal distribution, and that smoothness is what has made it the standard activation in transformers. Understanding its formula `x · Φ(x)` gives you a solid basis for reading the architecture of BERT or GPT and for choosing between activations with judgement. The natural next step is to see how the final output becomes probabilities with [the softmax function](/en/softmax-function-activation-for-classification/).

Sources: [1] [Gaussian Error Linear Units, Hendrycks and Gimpel (arXiv, 2016)](https://arxiv.org/abs/1606.08415), [2] [Activation function (Wikipedia)](https://en.wikipedia.org/wiki/Activation_function), [3] [torch.nn.GELU (PyTorch documentation)](https://pytorch.org/docs/stable/generated/torch.nn.GELU.html), [4] [tf.nn.gelu (TensorFlow documentation)](https://www.tensorflow.org/api_docs/python/tf/nn/gelu).

Route: The Neuron and Activation Functions