The Hyperbolic Tangent: A Powerful Activation Function

The hyperbolic tangent (tanh) is an activation function that squashes any real value into the interval (-1, 1) with zero-centred output, which removes the systematic gradient bias seen with sigmoid. It is the standard activation inside the LSTM and GRU memory cells used in recurrent networks.

March 24, 2023 4 min 411 4.3

Artificial Intelligence

The Sigmoid Function: A Key Tool in Neural Networks

The sigmoid function maps any real number to a value between 0 and 1, which makes it the natural activation function for expressing probabilities in a neural network. It is differentiable across its whole domain, though it suffers from saturation and vanishing gradients in deep networks, so today it is mostly reserved for the output layer.

March 24, 2023 4 min 787 4.2