Learning path Intermediate
The Neuron and Activation Functions
From the artificial neuron to forward propagation: weights and biases, the full family of activation functions (step, sigmoid, ReLU, GELU, Swish, ELU, SELU, Softplus, Mish) and the gradient problems.
- 20 resources
- 1 views
- ~114 min
This path explains, step by step, how an artificial neuron turns numbers into decisions: what each activation function actually does and why picking the right one matters as much as the architecture itself. It is built for readers who already code and want to understand the math behind a neural network, not just use it as a black box.
What you’ll be able to do
By the end of the 20 lessons you will be able to compute a neuron’s output by hand, explain why a network without nonlinear functions collapses into a linear model, and choose confidently between ReLU, GELU, Swish or Mish depending on the problem at hand. The level is intermediate: come in with some basic algebra (vectors, weighted sums) and prior coding experience, though no machine learning background is required.
How the sequence builds
It starts with the perceptron and the weighted sum of weights and biases, the foundation of any neuron. From there comes the central question: what an activation function is and why it is needed. The path then works through the full family, from the classic step and sigmoid functions to tanh, ReLU and its variants (Leaky ReLU, GELU, Swish, ELU, SELU, Softplus, Mish) and softmax for classification. It closes with a comparative guide for choosing a function, the vanishing and exploding gradient problems, and ties everything together in forward propagation through a full multilayer network.
That is 20 stops, the longest path in the neural network math catalog, and each one starts from a concrete formula, not a vague intuition.
The Perceptron, the Artificial Neuron and Its Maths
The perceptron is the simplest artificial neuron: it takes several inputs, multiplies them by its weights, adds a bias and applies an activation function that decides between two outputs. Frank Rosenblatt introduced it in 1958, and it remains the basic building block of every modern neural network.
Weights, Biases and a Neuron’s Weighted Sum
In an artificial neuron, weights measure how important each input is and the bias shifts the result. The neuron multiplies each input by its weight, adds everything up and includes the bias to produce the weighted sum z = Wx + b, the number that then passes through the activation function.
What Is an Activation Function and Why It Is Needed
An activation function is the nonlinear operation each neuron applies to its weighted sum z to produce its output a equals f of z. Without it, stacking layers only chains linear transformations and the whole network collapses into a single one. That nonlinearity is what lets a network learn complex patterns.
The Step Function: An Essential Tool in Neural Networks
The step function, or Heaviside function, is the simplest activation function in a neural network: it maps any input to a binary output, 0 or 1, depending on whether it crosses a fixed threshold. It was the core decision mechanism of Rosenblatt's perceptron in 1958, but its derivative is zero almost everywhere, so modern networks use sigmoid or ReLU instead.
Linear Function: A Common Activation Function
The linear function f(x) = ax + b is the simplest activation a neural network can use: in its identity form, f(x) = x, it is the standard choice for the output layer in regression, because it does not bound the range of possible values. In hidden layers it fails, because composing several linear functions collapses the whole network into one equivalent linear layer.
The Sigmoid Function: A Key Tool in Neural Networks
The sigmoid function maps any real number to a value between 0 and 1, which makes it the natural activation function for expressing probabilities in a neural network. It is differentiable across its whole domain, though it suffers from saturation and vanishing gradients in deep networks, so today it is mostly reserved for the output layer.
The Hyperbolic Tangent: A Powerful Activation Function
The hyperbolic tangent (tanh) is an activation function that squashes any real value into the interval (-1, 1) with zero-centred output, which removes the systematic gradient bias seen with sigmoid. It is the standard activation inside the LSTM and GRU memory cells used in recurrent networks.
The Rectified Linear Unit (ReLU): An Essential Tool for Deep Learning
ReLU (Rectified Linear Unit, f(x) = max(0, x)) has been the dominant activation function in deep learning since AlexNet popularised it in 2012: cheap to compute, resistant to vanishing gradients, and with one well-known weakness, dying ReLU.
The Leaky ReLU Function and Its Role in Neural Networks
Leaky ReLU is an activation function derived from ReLU that replaces the zero output for negative inputs with a small slope, typically 0.01. This keeps the gradient from ever reaching zero, which prevents the dying neuron problem and stabilizes training in very deep convolutional, recurrent, and GAN networks.
Softmax Function: Activation for Classification
The Softmax function transforms a neural network's output logits into a probability distribution that sums to 1. It is the standard activation for multi-class classification, from image classifiers to the vocabulary layer of language models like GPT.
The GELU Activation Function in Neural Networks
The GELU (Gaussian Error Linear Unit) function multiplies each input by the probability that a standard normal falls below that input. The result is a smooth curve with a continuous derivative that weights inputs by their magnitude, and it has become the default activation inside BERT and GPT.
The Swish (SiLU) Activation Function
The Swish function, also called SiLU, multiplies the input by its sigmoid: swish(x) = x·σ(x). It is smooth, non-monotonic and self-gating, so it often beats ReLU in deep networks. Models like LLaMA and EfficientNet use it as their default activation.
The ELU (Exponential Linear Unit) Activation Function
The ELU (Exponential Linear Unit) activation function returns x for positive inputs and α(eˣ−1) for negative ones. By allowing smooth negative values, it pushes the mean of the activations toward zero, speeds up convergence compared with ReLU, and avoids dead neurons thanks to a gradient that never vanishes completely on the negative side.
The SELU Activation and Self-Normalizing Networks
The SELU (Scaled Exponential Linear Unit) function is defined as SELU(x) equal to lambda times ELU(x), with lambda near 1.0507 and alpha near 1.6733. Those two constants make activations converge on their own towards zero mean and unit variance layer after layer, building deep networks that normalize themselves without batch normalization.
The Softplus Activation Function
The Softplus function is defined as softplus(x) = ln(1 + eˣ): a smooth, always-positive approximation of ReLU whose derivative is exactly the sigmoid. It is differentiable across its whole domain, avoids ReLU's angular kink and its output never quite reaches zero.
The Mish Activation Function
The Mish function is defined as mish(x) = x·tanh(softplus(x)). It is smooth, non-monotonic and self-regularized, properties that let it outperform Swish and ReLU across many tests. The YOLOv4 object detector adopted it in its backbone as the default activation.
How to Choose an Activation Function (Comparison)
Choosing an activation function is simple with one base rule: use ReLU in the hidden layers, GELU or SiLU in transformers, and reserve the output for softmax in multiclass classification, sigmoid in binary problems and a linear activation in regression. This comparison gathers formulas, ranges and use cases.
The Vanishing Gradient Problem
The vanishing gradient problem appears when the error signal shrinks as it backpropagates through many layers. The sigmoid and hyperbolic tangent functions have small derivatives, and their product tends toward zero, so the early layers barely learn. ReLU, careful initialisation and normalisation solve the problem.
The Exploding Gradient Problem
The exploding gradient problem happens when the gradient norm grows without control during backpropagation, especially in deep and recurrent networks with large weights. Training destabilises and the loss turns into NaN. Gradient clipping, together with good initialisation and normalisation, is the standard fix used today.
Forward Propagation in a Multilayer Network
Forward propagation is the process by which a neural network turns its input into a prediction, one layer at a time. Each layer multiplies the input vector by a weight matrix, adds a bias and applies an activation function, so the output of one layer feeds the next until the final result appears at the end.