The Swish (SiLU) Activation Function
Table of contents
The Swish function, also known as SiLU, is defined as swish(x) = x·σ(x): the input multiplied by its own sigmoid. It was born in 2017 out of an automated search for activation functions and has since become the default choice in architectures like LLaMA and EfficientNet. It is smooth, non-monotonic and self-gating, three properties that explain why it often outperforms ReLU in deep networks. This guide takes it apart piece by piece. The same explanation is available in Spanish.
Key takeaways
- The Swish activation function is defined as
swish(x) = x·σ(x), whereσis the sigmoid; that is why it is also called SiLU (Sigmoid Linear Unit). - It is smooth and differentiable across its whole domain, unlike ReLU, which has a non-differentiable kink at 0.
- It is non-monotonic: it has a minimum near -0.278 around
x = -1.278, a dip that ReLU lacks. - The self-gating mechanism lets the input itself modulate how much of itself it passes through.
- Modern models such as LLaMA (in its SwiGLU variant) and EfficientNet adopt it as their default activation.
What is Swish (SiLU)?
Swish is an activation function: the nonlinear operation applied to a neuron’s weighted sum z = Wx + b to produce its output a = f(z). Without that nonlinearity, stacking layers would be pointless, because the composition of linear functions is still linear.
The story has two almost simultaneous origins in 2017. The Google Brain team (Ramachandran, Zoph and Le) discovered it through an automated search over thousands of candidates and named it Swish. Independently, Elfwing, Uchibe and Doya proposed it in the context of reinforcement learning under the name SiLU. Today both names denote the same curve: x·σ(x).
The general form allows a parameter β: swish(x) = x·σ(βx). With β = 1 you recover the standard SiLU; as β grows, the function looks more and more like ReLU, and with β = 0 it collapses to the linear function x/2. In practice, most implementations fix β = 1.
Formula and derivative
We start from the sigmoid, $\sigma(x) = \dfrac{1}{1 + e^{-x}}$, which we analysed in detail in the sigmoid function. Swish uses it as a gate:
$$\operatorname{swish}(x) = x\,\sigma(x) = \dfrac{x}{1 + e^{-x}}$$
Its derivative has a compact and elegant form. Using the product rule and the property $\sigma'(x) = \sigma(x)\bigl(1-\sigma(x)\bigr)$, you get:
$$\operatorname{swish}'(x) = \sigma(x) + x\,\sigma(x)\bigl(1-\sigma(x)\bigr) = \operatorname{swish}(x) + \sigma(x)\bigl(1-\operatorname{swish}(x)\bigr)$$
Unlike ReLU, the derivative of Swish is continuous across its whole domain, including the origin. That smoothness is what sustains gradient descent in very deep networks and avoids the angular fold ReLU has at zero.
Three numerical consequences are worth remembering. For very positive values, $\sigma(x)$ tends to 1 and Swish behaves almost like the identity $x$. For very negative values, it tends to 0 smoothly, without cutting off abruptly like ReLU. And around $x = -1.278$ it reaches its global minimum, roughly $-0.278$: that is where the characteristic dip appears.
See the derivation of the derivative
Applying the product rule to $x\,\sigma(x)$: the first term differentiates $x$ and leaves $\sigma(x)$; the second keeps $x$ and differentiates $\sigma(x)$, whose derivative is $\sigma(x)\bigl(1-\sigma(x)\bigr)$. Adding the two: $\operatorname{swish}'(x) = \sigma(x) + x\,\sigma(x)\bigl(1-\sigma(x)\bigr)$, which rewrites as $\operatorname{swish}(x) + \sigma(x)\bigl(1-\operatorname{swish}(x)\bigr)$.
Self-gating and smoothness
The key term for understanding Swish is self-gating. In gated architectures, such as LSTMs, an external value decides how much information passes. In Swish, the gate σ(x) is computed by the input itself: x decides how much of itself it lets flow. When x is large, the gate opens and passes almost everything; when it is very negative, the gate closes smoothly.
Smoothness matters for two reasons. First, the gradient is continuous everywhere, which helps optimisation based on derivatives and avoids ReLU’s angular fold at the origin. Second, by allowing a small negative flow, Swish does not suffer from ReLU’s "dying neuron" problem, where a neuron that only receives negative inputs stops learning because its gradient is stuck at 0. The original paper by Ramachandran, Zoph and Le, Searching for Activation Functions[1], argues that these properties together, not any single one, explain the observed improvement.
Use in LLaMA and modern models
Swish stopped being an academic curiosity when it entered reference models. EfficientNet, Google’s 2019 family of vision networks, uses it as the default activation in all its blocks and with it reached the state of the art on ImageNet with fewer parameters than its competitors.
In large language models, the most influential variant is SwiGLU, a gated unit that combines Swish with the GLU mechanism. Meta’s LLaMA adopts it in the layers of its feed-forward network, following the line of Noam Shazeer’s work, GLU Variants Improve Transformer[2]. Current frameworks give you the function ready to use: in PyTorch it is torch.nn.SiLU[3], and the name SiLU, not Swish, became the standard in the documentation.
Swish versus ReLU
ReLU, max(0, x), is still the most widely used activation for its simplicity and low cost. So why bother with Swish? The difference shows up mainly in very deep networks. In their experiments, the authors observed improvements of around 0.6% to 0.9% in ImageNet top-1 accuracy when replacing ReLU with Swish in networks like Mobile NASNet-A and Inception-ResNet-v2, changing nothing else.
The cost is real: each Swish evaluation involves computing an exponential and a multiplication, while ReLU is a simple comparison. That is why ReLU remains preferable when latency rules or the network is shallow, and Swish gains ground when depth and final quality matter more than a few milliseconds per layer. You can place both within the full route in the mathematics behind neural networks roadmap.
Frequently asked questions
Are Swish and SiLU the same thing?
Yes. SiLU (Sigmoid Linear Unit) was the name given by Elfwing and colleagues; Swish is the one chosen by the Google Brain team. Both describe exactly the same function x·σ(x). In the PyTorch and TensorFlow documentation you will mostly see the SiLU name.
When should I use Swish instead of ReLU?
It is worth it in deep networks where final accuracy is a priority, such as large vision or language models. If the network is small or inference speed is critical, ReLU is usually enough and considerably cheaper to compute.
Why is Swish said to be non-monotonic?
Because it does not increase steadily: for negative inputs the function dips to a minimum near -0.278 before rising again. That small dip lets it represent relationships that an always-increasing function, like ReLU, does not capture equally well.
Conclusion
The Swish function distils a powerful idea into a tiny formula: let the input regulate its own passage with x·σ(x). Smoothness, non-monotonicity and self-gating combine to beat ReLU in the most demanding architectures, from EfficientNet to LLaMA. The natural next step is to compare it with the other activations and with the sigmoid function that serves as its gate.
