Categories
Beginner

Neural network math: activation functions

Activation functions are what let a neural network learn non-linear relationships. This path walks through them in teaching order, from the simplest (the step function) to today’s most used (ReLU and softmax), each with its formula, derivative and interactive plots so you actually understand them.

  • 7 resources
  • ~31 min read

  1. The Step Function: An Essential Tool in Neural Networks

    The step function, or Heaviside function, is the simplest activation function in neural networks: it converts any numeric input into a binary output, 0 or 1, depending on whether it crosses a fixed threshold. It was the central mechanism of Rosenblatt's 1958 perceptron, but because it is not differentiable, it cannot be used in modern backpropagation training.

    • 3 min

  2. Linear Function: A Common Activation Function

    The linear function, f(x) = ax + b, is the simplest activation a neural network can use: its output is directly proportional to the input, with no non-linear transformation. It is the standard choice for the output layer in regression problems, but in hidden layers it collapses the entire network into a single linear model, so it should never be used there.

    • 5 min

  3. The Sigmoid Function: A Key Tool in Neural Networks

    The sigmoid function compresses any real value into the range (0, 1), making it the natural activation function for modelling probabilities in neural networks. It is differentiable everywhere, enabling training via backpropagation, though it suffers from saturation and vanishing gradients in deep layers, where ReLU and tanh have taken over.

    • 5 min

  4. The Hyperbolic Tangent: A Powerful Activation Function

    The hyperbolic tangent (tanh) is an activation function that maps any real value to the interval (-1, 1) with zero-centred output, which makes it more stable than sigmoid in hidden layers. It is the standard in LSTM and GRU memory cells, though it shares with sigmoid the vanishing-gradient problem at extreme inputs.

    • 6 min

  5. The Rectified Linear Unit (ReLU): An Essential Tool for Deep Learning

    ReLU (rectified linear unit) is the most widely used activation function in deep neural networks: it returns the input value if positive, and zero if negative, defined as f(x) = max(0, x). Its low computational cost and resistance to the vanishing gradient problem that hampers sigmoid made it the de facto standard since AlexNet in 2012.

    • 4 min

  6. The Leaky ReLU Function and Its Role in Neural Networks

    Leaky ReLU is a variant of the ReLU function that replaces zero for negative values with a small slope, keeping neurons from ever fully shutting down. This solves the dying neuron problem and improves training stability in deep neural networks, CNNs, and GAN discriminators.

    • 4 min

  7. Softmax Function: Activation for Classification

    The Softmax function converts a vector of logits (arbitrary values) into a probability distribution where every value is positive and the values sum to exactly 1. It is the standard output-layer activation for multi-class classification, and the final operation language models use to predict the next token.

    • 4 min