The Softplus Activation Function
Table of contents
The Softplus function is defined as softplus(x) = ln(1 + eˣ): a smoothed version of ReLU whose derivative is, exactly, the sigmoid function. It was proposed by Dugas and colleagues in 2001 and has served ever since as the textbook example of an activation that is differentiable across its whole domain. It always returns a positive value, bends gradually instead of breaking at a corner, and never suffers ReLU’s abrupt cut-off at the origin. This guide takes it apart piece by piece. The same explanation is available in Spanish.
Key takeaways
- The Softplus activation function is defined as
softplus(x) = ln(1 + eˣ), a smooth approximation of ReLU introduced by Dugas and others in 2001. - Its derivative is exactly the sigmoid,
σ(x) = 1 / (1 + e⁻ˣ), which guarantees a continuous gradient bounded between 0 and 1. - Its output is always strictly positive: the range is
(0, ∞)and at the origin it equalsln(2) ≈ 0.693, not zero. - It is differentiable everywhere, without the angular kink ReLU has at
x = 0. - In practice ReLU usually matches or beats it, so Softplus is reserved for cases where forcing positive, smooth outputs matters.
What is Softplus?
Softplus is an activation function: the nonlinear operation applied to a neuron’s weighted sum z = Wx + b to produce its output a = f(z). Without that nonlinearity, stacking layers would add nothing, because the composition of linear functions is still linear.
The name captures its intent. It is a «soft» version of the plus function, that is, of the positive part max(0, x) that defines the rectified unit ReLU. Where ReLU cuts off sharply at the origin, Softplus describes the same overall trend with a continuous curve that bends slowly. Dugas and colleagues introduced it in the paper Incorporating Second-Order Functional Knowledge for Better Option Pricing[1], where they needed an increasing, smooth function to model option prices.
Formula and derivative (the sigmoid)
The definition is a single line:
$$\operatorname{softplus}(x) = \ln(1 + e^{x})$$
Three concrete values are worth remembering. At $x = 0$, the function equals $\ln(1 + 1) = \ln(2) \approx 0.693$. For very negative values, $e^{x}$ tends to 0 and the function approaches 0 smoothly without ever touching it. For very positive values, $e^{x}$ dominates and $\ln(1 + e^{x}) \approx x$, so the curve hugs the identity line asymptotically. That is the key to its resemblance to ReLU.
The most elegant part appears on differentiation. The derivative of $\ln(1 + e^{x})$ is $e^{x} / (1 + e^{x})$, which rewritten is $1 / (1 + e^{-x})$, that is, the sigmoid function:
$$\operatorname{softplus}'(x) = e^{x} / (1 + e^{x}) = 1 / (1 + e^{-x}) = \sigma(x)$$
Show the derivation
We differentiate $\ln(1 + e^{x})$ with the chain rule. If $u = 1 + e^{x}$, then $u’ = e^{x}$ and $\frac{d}{dx}\ln(u) = u’/u = e^{x}/(1 + e^{x})$. Dividing numerator and denominator by $e^{x}$ gives $1/(1 + e^{-x}) = \sigma(x)$, the sigmoid.
Because the derivative of Softplus is the sigmoid, its gradient is never exactly zero: that is why, unlike ReLU, no Softplus neuron ever fully “dies” during training.
Put the other way round, Softplus is the antiderivative of the sigmoid. The gradient then inherits $\sigma$’s properties: it is always positive, continuous and bounded in the interval $(0, 1)$, which provides a stable gradient flow during backpropagation.
Softplus as a smooth ReLU
Comparing the two curves clarifies what Softplus gains and loses. ReLU is max(0, x): exactly 0 for negative inputs and the identity for positive ones, with a non-differentiable fold at the origin. Softplus draws that same silhouette but rounded. The largest difference between the two functions occurs precisely at x = 0, where the gap equals ln(2) ≈ 0.693, and it narrows quickly as we move away: at x = 4 the separation is already below 0.02.
There is a version with a temperature parameter, softplus(x) = (1/β)·ln(1 + eᵝˣ). As β grows, the curve clings ever more tightly to ReLU; in the limit β → ∞ you recover exactly max(0, x). As Goodfellow, Bengio and Courville summarise in Deep Learning[2], «the use of the softplus is generally discouraged», because, despite being differentiable everywhere, in practice it does not improve on the rectifier. Smoothness, which looked like an obvious advantage, does not translate into better empirical performance.
Advantages and drawbacks
Softplus has three advantages, all stemming from its smoothness. First, it is differentiable across all of ℝ, without ReLU’s angular point, which can suit second-order optimisation methods that need continuous curvature. Second, its output is always strictly positive, a very useful property when the network must predict a quantity that cannot be negative, such as a variance, a rate or a scale parameter. Third, it does not suffer from the "dying neuron" problem: because the gradient σ(x) is never exactly 0, no neuron stops learning entirely.
The drawbacks explain why it never dethroned ReLU. The cost is higher, since each evaluation requires an exponential and a logarithm, compared with ReLU’s simple comparison. Furthermore, by not producing exact zeros, Softplus does not generate the sparse representation that makes ReLU attractive. And, as Glorot, Bordes and Bengio showed in Deep Sparse Rectifier Neural Networks[3], the rectifier matches or beats Softplus on the classification tasks they tested, despite not being differentiable at the origin.
Stable implementation
The naive formula ln(1 + exp(x)) is a numerical trap. For large x, exp(x) overflows: in 32-bit floating point the result is already infinity around x = 88, and then ln(inf) returns infinity instead of the correct value, which is roughly x. The standard solution rewrites the function in an equivalent but safe form:
$$\operatorname{softplus}(x) = \max(0, x) + \ln(1 + e^{-|x|})$$
This way the exponential’s argument is never positive, e^-|x| always stays in (0, 1] and there is no overflow. Frameworks also apply a threshold: PyTorch implements torch.nn.Softplus[4] with the parameters beta=1 and threshold=20, so above that point it replaces the formula with the line x, by then indistinguishable from the real curve and with no risk of losing precision. If you code the function by hand, start from the stable variant and not from the direct definition.
Frequently asked questions
What is the derivative of Softplus?
The derivative of softplus(x) = ln(1 + eˣ) is the sigmoid function, σ(x) = 1 / (1 + e⁻ˣ). Put another way, Softplus is the antiderivative or integral of the sigmoid. That is why its gradient is continuous, always positive and bounded between 0 and 1.
How does Softplus differ from ReLU?
Both share the same overall shape, but ReLU cuts off sharply at 0 with a non-differentiable corner, while Softplus approximates it with a smooth curve. Softplus is never exactly zero (at the origin it equals ln(2) ≈ 0.693) and is more expensive to compute, since it needs an exponential and a logarithm.
When should I use Softplus?
It is useful when the network must emit a value that is necessarily positive, such as a variance or a rate, or when the optimisation algorithm requires a function differentiable across its whole domain. For general-purpose hidden layers, ReLU or its variants usually perform as well or better at lower cost.
Conclusion
The Softplus function distils a clear idea: smooth ReLU with ln(1 + eˣ) so the curve bends instead of breaking and its derivative is the sigmoid. That refinement makes it differentiable everywhere and always positive, but it rarely beats the rectifier in practice, so its place is in cases where positivity or smoothness is a requirement. You can place it alongside the other activations within the mathematics behind neural networks roadmap.
