The Mish Activation Function
The Mish function is defined as mish(x) = x·tanh(softplus(x)): the input multiplied by the hyperbolic tangent of its softplus. Diganta Misra proposed it in 2019 as a smooth, non-monotonic and self-regularized activation, in the same family as Swish. Its moment of fame came in 2020, when the YOLOv4 object detector chose it for its backbone. This guide takes it apart piece by piece: formula, derivative, properties and cost. The same explanation is available in Spanish.
Key takeaways
- The Mish activation function is defined as
mish(x) = x·tanh(softplus(x)), wheresoftplus(x) = ln(1 + e^x). - It is smooth and differentiable across its whole domain, unlike ReLU, which has a non-differentiable kink at 0.
- It is non-monotonic: it reaches a global minimum near -0.31 around
x = -1.19, a dip similar to Swish’s. - It is bounded below and unbounded above, which produces a regularizing effect without extra dropout.
- The YOLOv4 detector adopted it in its CSPDarknet53 backbone and with it reached 43.5% AP on the COCO dataset.
What is Mish?
Mish is an activation function: the nonlinear operation applied to a neuron’s weighted sum z = Wx + b to produce its output a = f(z). Without that nonlinearity, stacking layers would be pointless, because the composition of linear functions is still linear.
Diganta Misra introduced it in 2019 in the paper Mish: A Self Regularized Non-Monotonic Activation Function[1]. The author describes it as «a smooth, continuous, self-regularized, non-monotonic activation function», four adjectives that capture its whole appeal. Its shape is deliberately close to Swish, but it swaps the sigmoid gate for the combination of softplus and hyperbolic tangent.
The name softplus denotes a smoothed version of ReLU: softplus(x) = ln(1 + e^x). When you pass it through the hyperbolic tangent and multiply by the input, you get Mish’s characteristic curve, almost linear for large values and with a smooth dip in the negative region.
Formula and derivative
The compact definition chains three well-known functions:
$$\operatorname{mish}(x) = x \cdot \tanh(\operatorname{softplus}(x)) = x \cdot \tanh(\ln(1+e^x))$$
Its derivative follows from the product and chain rules. If we let $\operatorname{sp}(x) = \operatorname{softplus}(x)$ and recall that $\operatorname{sp}'(x) = \sigma(x)$ (the sigmoid) and that the derivative of the hyperbolic tangent is $\operatorname{sech}^2$, we get:
$$\operatorname{mish}'(x) = \tanh(\operatorname{sp}(x)) + x \cdot \sigma(x) \cdot \operatorname{sech}^2(\operatorname{sp}(x))$$
Mish is smooth and differentiable across its whole domain: its gradient is continuous even at the origin, without the “kink” that ReLU has at zero, where its derivative jumps abruptly from 0 to 1.
Three numerical consequences are worth remembering. For very positive values, $\tanh(\operatorname{softplus}(x))$ tends to 1 and Mish behaves almost like the identity $x$. For very negative values, it tends to 0 smoothly, without cutting off abruptly like ReLU. And around $x=-1.19$ it reaches its global minimum, roughly $-0.31$: that is where the dip that makes it non-monotonic appears.
Show the derivation
Start from $\operatorname{mish}(x) = x \cdot \tanh(\operatorname{sp}(x))$ and apply the product rule: the first term differentiates $x$ and leaves $\tanh(\operatorname{sp}(x))$; the second keeps $x$ and differentiates $\tanh(\operatorname{sp}(x))$. Since $\frac{d}{dx}\tanh(u) = \operatorname{sech}^2(u)\cdot u’$ and here $u=\operatorname{sp}(x)$ with $\operatorname{sp}'(x) = \sigma(x)$, the chain rule gives $\operatorname{sech}^2(\operatorname{sp}(x))\cdot\sigma(x)$. Adding both terms: $\operatorname{mish}'(x) = \tanh(\operatorname{sp}(x)) + x\cdot\sigma(x)\cdot\operatorname{sech}^2(\operatorname{sp}(x))$.
Smoothness and non-monotonicity
Smoothness matters for two reasons. First, the gradient is continuous everywhere, which helps optimisation based on derivatives and avoids ReLU’s angular fold at the origin. A smoother error surface makes it easier for gradient descent to find good minima.
Second, by allowing a small negative flow, Mish does not suffer from ReLU’s "dying neuron" problem, where a neuron that only receives negative inputs stops learning because its gradient is stuck at 0. The non-monotonicity (that dip below zero) preserves small but useful gradients for negative inputs.
There is also a property that sets it apart: Mish is bounded below, with that floor near -0.31, but unbounded above. Being bounded below introduces a regularizing effect, while being unbounded above avoids the saturation that stalls learning in functions like the sigmoid. That is why Misra calls it a "self-regularized" activation.
Performance versus Swish and ReLU
The practical question is whether that elegance translates into better results. In the original paper, Misra tested Mish on more than 70 benchmark problems against other activations. One of the most cited figures: on CIFAR-100, with an 18-layer Squeeze Excite Network, Mish improved top-1 accuracy by 0.494% over Swish and by 1.671% over ReLU, changing nothing else in the network.
The practical breakthrough came with computer vision. YOLOv4, presented in 2020 by Bochkovskiy, Wang and Liao in YOLOv4: Optimal Speed and Accuracy of Object Detection[2], uses Mish in its CSPDarknet53 backbone and it is part of the "bag of improvements" with which the model reached 43.5% AP on COCO at real-time speed. Against Swish, the differences are usually small: both share a shape and its advantages, and in many cases the choice depends more on the compute budget than on accuracy.
Computational cost
Here is the trade-off. Each Mish evaluation chains an exponential (for softplus), an implicit logarithm, a hyperbolic tangent and a multiplication, while ReLU is a simple comparison with zero and Swish needs a single sigmoid. In practice, Mish is the most expensive of the three both in compute and in memory during backpropagation.
That cost can be trimmed. A fused implementation, Mish-CUDA, combines the operations in a single GPU kernel and cuts memory use and time per layer. Even so, when latency rules or the network is shallow, ReLU remains the sensible choice. Current frameworks give you the function ready to use: in PyTorch it is torch.nn.Mish[3]. You can place Mish, Swish and ReLU within the full route in the mathematics behind neural networks roadmap.
Frequently asked questions
How does Mish differ from Swish?
Both are smooth, non-monotonic and self-gating, and their curves look very similar. The difference is in the gate: Swish uses the sigmoid (x·σ(x)), while Mish uses tanh(softplus(x)). In performance they are usually very close; Mish is somewhat more expensive to compute, but in some tests it offers a marginal improvement.
Why is Mish said to be self-regularized?
Because it is bounded below (with a floor near -0.31) but unbounded above. That lower limit introduces a regularizing effect on the activations, similar in spirit to what techniques like dropout aim for, without adding explicit noise during training.
When should I use Mish?
It is worth it in deep networks where final accuracy is a priority, such as object detectors or large vision networks. If the network is small or inference speed is critical, ReLU is usually enough and considerably cheaper to compute.
Conclusion
The Mish function distils a powerful idea into a compact formula: let the input pass through a smooth gate made of softplus and hyperbolic tangent with x·tanh(softplus(x)). Smoothness, non-monotonicity and self-regularization combine to match or beat Swish and ReLU in the most demanding architectures, from CIFAR-100 to YOLOv4. The natural next step is to compare it with the Swish (SiLU) function, its closest relative.
