Three simple matrix operations hold up much of the maths inside a neural network: the transpose, the identity matrix and the inverse. The transpose rearranges a table of numbers, the identity acts as the neutral element and the inverse undoes what another matrix did. With these three pieces you can follow backpropagation and the solving of linear systems. This guide explains them one by one, with numeric examples and nothing taken for granted. The same explanation is available in Spanish.

Key formula $(\mathbf{A}^\top)_{ij} = \mathbf{A}_{ji}$

Key takeaways

  • The transpose and inverse matrix are two different operations: the transpose swaps rows and columns, while the inverse undoes the transformation a matrix performs.
  • The transpose of a 2×3 matrix is a 3×2 matrix; it is written Aᵀ and satisfies (AB)ᵀ = BᵀAᵀ.
  • The identity matrix I has ones on the diagonal and zeros elsewhere; multiplying by it changes nothing, just like multiplying a number by 1.
  • The inverse A⁻¹ exists only if the matrix is square and its determinant is not zero; then A·A⁻¹ = I.
  • In backpropagation, the gradient flowing back through a layer uses the transpose of the weight matrix Wᵀ.

The transpose and its role in backprop

The transpose of a matrix A is another matrix, written Aᵀ, obtained by swapping its rows for its columns. The element in row i and column j of A moves to row j and column i of Aᵀ. That is why a 2×3 matrix (2 rows, 3 columns) becomes a 3×2 matrix once transposed.

This operation is not a decorative technicality. In forward propagation a layer computes z = Wx + b, the same expression we study in the mathematics of neural networks roadmap. When the error flows back during backpropagation, the gradient with respect to the layer’s input is computed by multiplying by Wᵀ, the transpose of the weight matrix. The transpose is what makes the dimensions fit: if W turns an input vector into an output vector, Wᵀ carries the gradient the other way, from output to input.

Two properties come in handy when coding: the transpose of a transpose returns the original matrix, $(\mathbf{A}^\top)^\top = \mathbf{A}$, and the transpose of a product reverses the order, $(\mathbf{A}\mathbf{B})^\top = \mathbf{B}^\top\mathbf{A}^\top$. That last rule is what makes the dimensions of backpropagation line up layer by layer.

The identity matrix

The identity matrix, written I, is the square matrix with ones on the main diagonal and zeros in every other position. A 3×3 identity therefore has 3 ones and 6 zeros. Its role is that of the neutral element of matrix multiplication: for any compatible matrix A we have $\mathbf{A}\mathbf{I} = \mathbf{I}\mathbf{A} = \mathbf{A}$.

As Goodfellow, Bengio and Courville define it in their book Deep Learning, «an identity matrix is a matrix that does not change any vector when we multiply that vector by that matrix». That behaviour makes it the reference against which the inverse is defined, and a convenient starting point for initialising certain layers, such as some residual connections, where you want to begin without altering the signal.

The inverse matrix and when it exists

The inverse of a matrix A, written $\mathbf{A}^{-1}$, is the matrix that undoes its effect: multiplying them gives the identity.

$$\mathbf{A}\mathbf{A}^{-1} = \mathbf{A}^{-1}\mathbf{A} = \mathbf{I}$$

If a matrix represents a transformation (rotate, scale, project), its inverse is the transformation that returns each vector to its starting point.

Not every matrix has an inverse. For A⁻¹ to exist two conditions must hold: the matrix has to be square (the same number of rows as columns) and its determinant cannot be zero. When the determinant is 0 the matrix is called singular and squashes the space onto a lower dimension, so information is lost and there is no way back. A 2×2 matrix such as [[1, 2], [2, 4]] has determinant 0 (because 1·4 − 2·2 = 0) and is therefore not invertible.

In deep learning practice a weight matrix is rarely inverted explicitly: it is costly and numerically unstable. Instead we solve linear systems or use gradient descent. The inverse matters mainly as a concept and appears in classic methods such as least-squares regression and in second-order optimisation techniques.

In an orthogonal matrix the inverse equals the transpose, $\mathbf{A}^{-1} = \mathbf{A}^\top$. That is why inverting a rotation is as cheap as transposing it, and why some layers are initialised with orthogonal matrices to preserve the signal.

Use in neural networks

These three operations combine constantly. In backpropagation the transpose of the weight matrix redirects the gradient towards earlier layers. The identity appears when initialising weights and in the residual connections of architectures like ResNet, which added millions of effective layers by letting the signal pass through unchanged. And the idea of an inverse is behind the solving of systems and behind why we prefer iterative methods: inverting a 1000×1000 matrix takes on the order of a billion operations, whereas a gradient-descent step is far cheaper.

If you want to see how they fit with the matrix product inside each layer, the natural next step is matrix multiplication in neural networks.

Worked examples

Transposing a matrix

We start from a 2×3 matrix:

$$\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \ 4 & 5 & 6 \end{bmatrix}$$

Its transpose is 3×2, with rows turned into columns:

$$\mathbf{A}^\top = \begin{bmatrix} 1 & 4 \ 2 & 5 \ 3 & 6 \end{bmatrix}$$

In NumPy this is simply A.T, an operation that does not even copy the data: it only changes how the array is traversed.

Checking an inverse

We take a square matrix with a non-zero determinant:

$$\mathbf{A} = \begin{bmatrix} 2 & 0 \ 0 & 4 \end{bmatrix}$$

Its determinant is 2·4 − 0·0 = 8, so it is invertible. The inverse is obtained by inverting each value on the diagonal:

$$\mathbf{A}^{-1} = \begin{bmatrix} 0.5 & 0 \ 0 & 0.25 \end{bmatrix}$$

Multiplying them back recovers the identity, $\mathbf{A}\mathbf{A}^{-1} = \begin{bmatrix} 1 & 0 \ 0 & 1 \end{bmatrix} = \mathbf{I}$, which is the check that the calculation is right.

Frequently asked questions

What is the difference between the transpose and the inverse of a matrix?

The transpose only rearranges the numbers, swapping rows for columns, and exists for any matrix. The inverse, by contrast, undoes the transformation the matrix represents and only exists if the matrix is square and its determinant is not zero. They serve different purposes, though in orthogonal matrices they coincide: there the transpose equals the inverse.

Why does the transpose appear in backpropagation?

Because the gradient travels the opposite way to the signal. If a layer transforms the input with matrix W, the error has to flow from output back to input, and the matrix that performs that reverse path, respecting the dimensions, is the transpose Wᵀ. It is not the inverse: it is the transpose, which is far cheaper to compute and always exists.

Does every matrix have an inverse?

No. Only square matrices with a non-zero determinant, called invertible or non-singular, have an inverse. If the determinant is 0 the matrix collapses the space and loses information, so there is no way to undo its effect.

Conclusion

The transpose, the identity matrix and the inverse are the minimum vocabulary for reading a neural network’s formulas fluently. The transpose rearranges and moves gradients backwards, the identity is the 1 that changes nothing, and the inverse undoes transformations when it exists. With these three ideas clear, backpropagation stops being a mysterious formula. The next step is to review the full mathematics of neural networks roadmap and continue with matrix multiplication.

Sources

  1. Mathematics for Machine Learning (Deisenroth, Faisal and Ong)
  2. Deep Learning (Goodfellow, Bengio and Courville)
  3. numpy.transpose, NumPy documentation
  4. Invertible matrix (Wikipedia)

Route: Mathematical Foundations of Neural Networks