Technology
The Swish (SiLU) Activation Function
The Swish function, also called SiLU, multiplies the input by its sigmoid: swish(x) = x·σ(x). It is smooth, non-monotonic and self-gating, so it often beats ReLU in deep networks. Models like LLaMA and EfficientNet use it as their default activation.