Technology
The Softplus Activation Function
The Softplus function is defined as softplus(x) = ln(1 + eˣ): a smooth, always-positive approximation of ReLU whose derivative is exactly the sigmoid. It is differentiable across its whole domain, avoids ReLU's angular kink and its output never quite reaches zero.