Forward Propagation in a Multilayer Network

Forward propagation is the journey a piece of data takes from the input to the output of a neural network, moving layer by layer. At each layer a vector is multiplied by a weight matrix, a bias is added and an activation function is applied. The output of one layer feeds the next until the final prediction appears. Understanding this flow is the step before understanding how the network learns. The same explanation is available in Spanish.

Key takeaways

• Forward propagation turns the input $\mathbf{x}$ into a prediction by applying, at each layer, the formula $\mathbf{a}^{(l)} = f!\left(\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}\right)$.
• Each layer performs three operations: a matrix-by-vector product, the bias addition and a nonlinear activation function.
• The superscript $(l)$ identifies the layer: $\mathbf{W}^{(1)}$ are the weights of the first layer and $\mathbf{a}^{(2)}$ the activation of the second.
• Without activation functions, stacking 2, 10 or 100 layers would collapse to a single linear transformation.
• The output is compared with the correct answer through a loss function, and that number is the starting point of backpropagation.

What is forward propagation?

Forward propagation (also called the forward pass) is the computation that carries an input example, through every layer, to the network’s output. No learning happens here: the weights are fixed and we only evaluate the function the network defines. It is one half of the training cycle; the other half, backpropagation, comes afterwards.

The starting point is an input vector $\mathbf{x}$, which we rename as the activation of layer zero: $\mathbf{a}^{(0)} = \mathbf{x}$. From there, each layer takes the previous activation, computes a weighted sum and applies a nonlinearity. That mechanism of weights, biases and weighted sums is covered in weights, biases and the weighted sum, and here we chain it layer after layer. To place this piece within the full route, the mathematics of neural networks roadmap shows where it fits.

Layer-by-layer computation with matrices

For a layer $l$ with previous activation $\mathbf{a}^{(l-1)}$, the computation is written in two steps:

$$\begin{aligned} \mathbf{z}^{(l)} &= \mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)} \quad \text{(weighted sum, linear result)}\ \mathbf{a}^{(l)} &= f!\left(\mathbf{z}^{(l)}\right) \quad \text{(activation, nonlinearity)} \end{aligned}$$

Here $\mathbf{W}^{(l)}$ is the layer’s weight matrix, $\mathbf{b}^{(l)}$ the bias vector and $f$ the activation function (ReLU, sigmoid or another). The dimensions must fit: if layer $l$ has 4 neurons and receives 3 inputs, then $\mathbf{W}^{(l)}$ is a $4\times3$ matrix, $\mathbf{b}^{(l)}$ a 4-component vector and $\mathbf{a}^{(l)}$ another 4-vector. That matrix-by-vector product is the central operation, the same one studied in matrix multiplication in neural networks, and the reason graphics cards speed up deep learning so much.

As Goodfellow, Bengio and Courville define it in their reference book, «deep feedforward networks are the quintessential deep learning models». Forward propagation is, literally, the definition of what the network computes.

Layer index notation

The parenthesised superscript indicates the layer, not a power. In a network of $L$ layers, $\mathbf{W}^{(1)}$ and $\mathbf{b}^{(1)}$ belong to the first hidden layer, and $\mathbf{W}^{(L)}$, $\mathbf{b}^{(L)}$ to the output layer. The input activation is $\mathbf{a}^{(0)} = \mathbf{x}$ and the final prediction is $\mathbf{a}^{(L)}$.

This convention lets us write the whole network as a single chain of compositions:

$$\mathbf{a}^{(L)} = f!\left(\mathbf{W}^{(L)}\cdots f!\left(\mathbf{W}^{(2)}f!\left(\mathbf{W}^{(1)}\mathbf{x}+\mathbf{b}^{(1)}\right)+\mathbf{b}^{(2)}\right)\cdots+\mathbf{b}^{(L)}\right)$$

The same notation appears in Michael Nielsen’s^[1] classic treatment and in the feedforward neural network^[2] article on Wikipedia. Keeping the indices ordered avoids the most common bug when coding a network: mismatching the dimensions of two consecutive matrices.

A complete numerical example

Take a small network with 2 inputs, a hidden layer of 2 neurons with ReLU activation and one output with sigmoid activation. We start from the input $\mathbf{a}^{(0)} = [1, 2]$.

Layer 1, with $\mathbf{W}^{(1)} = \begin{bmatrix}0.1 & 0.3\ 0.2 & 0.4\end{bmatrix}$ and $\mathbf{b}^{(1)} = [0.1,\, 0.2]$:

$$\begin{aligned} z_1 &= 0.1\cdot1+0.3\cdot2+0.1 = 0.8 \quad \text{(first neuron)}\ z_2 &= 0.2\cdot1+0.4\cdot2+0.2 = 1.2 \quad \text{(second neuron)}\ a_1 &= \operatorname{ReLU}(0.8)=0.8,\quad a_2=\operatorname{ReLU}(1.2)=1.2 \end{aligned}$$

The hidden-layer activation is therefore $\mathbf{a}^{(1)} = [0.8,\, 1.2]$. Now the output layer, with $\mathbf{W}^{(2)} = [0.5,\, -0.6]$ and $\mathbf{b}^{(2)} = 0.25$:

$$\begin{aligned} z^{(2)} &= 0.5\cdot0.8+(-0.6)\cdot1.2+0.25 = 0.4-0.72+0.25 = -0.07\ a^{(2)} &= \operatorname{sigmoid}(-0.07)\approx0.482 \end{aligned}$$

The network predicts $0.482$, a value between 0 and 1 we can read as a probability. The whole computation is 3 multiply-adds in the first layer, 2 in the second and two activations: nothing more than chained arithmetic. With these same steps, repeated millions of times, a model like GPT-3 evaluates its 175 billion parameters on every forward pass.

From forward propagation to the loss function

The prediction $\mathbf{a}^{(L)}$ on its own does not tell us whether the network is right. For that it is compared with the correct answer $y$ through a loss function $L$. In our example, if the correct label were $y = 1$, the binary cross-entropy would give $L = -\ln(0.482) \approx 0.729$: the further from 1, the larger the penalty.

That number is the hinge between the two halves of training. Forward propagation produces the loss; then backpropagation walks the network in reverse and computes the gradient $\nabla L$ with respect to each weight, so that gradient descent adjusts $\mathbf{W}$ and $\mathbf{b}$ with a learning rate $\eta$. We place that next step within the full mathematics-of-the-network roadmap.

Frequently asked questions

How is forward propagation different from backpropagation?

Forward propagation goes from input to output and computes the prediction and the loss with the current weights. Backpropagation goes from output to input and computes how to change those weights. One evaluates the network; the other trains it. They always run in that order: forward first, backward second.

Why do we need a nonlinear activation function?

Because without it each layer would just be $\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}$, a linear operation, and composing several linear operations gives another linear operation. Stacking 100 layers would add nothing: the network could not represent curved relationships. The nonlinearity of $f$ is what gives real depth to a deep network.

What dimensions must the weight matrices have?

If layer $l$ produces $n$ outputs from $m$ inputs, $\mathbf{W}^{(l)}$ is an $n\times m$ matrix, $\mathbf{b}^{(l)}$ a vector of $n$ and $\mathbf{a}^{(l)}$ another vector of $n$. The practical rule: the number of columns of $\mathbf{W}^{(l)}$ must match the number of components of $\mathbf{a}^{(l-1)}$.

Conclusion

Forward propagation is the part of the network many people already grasp without naming it: multiply by some weights, add a bias, activate and repeat. Written with matrices and the index notation $(l)$, it fits in a single line, $\mathbf{a}^{(l)} = f!\left(\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)}\right)$, and generalises to any number of layers. The next step is to close the cycle with the loss function and backpropagation, where the network truly begins to learn.

Sources

1. Michael Nielsen’s
2. feedforward neural network
3. Deep Learning (Goodfellow, Bengio and Courville), chapter 6
4. Mathematics for Machine Learning (Deisenroth, Faisal and Ong)

Route: The Neuron and Activation Functions