Technology
The GELU Activation Function in Neural Networks
The GELU (Gaussian Error Linear Unit) function multiplies each input by the probability that a standard normal falls below that input. The result is a smooth curve with a continuous derivative that weights inputs by their magnitude, and it has become the default activation inside BERT and GPT.