Tag
The transpose swaps rows for columns and shows up at every step of backpropagation; the identity matrix acts as the 1 of matrix algebra and leaves any vector unchanged, and the inverse matrix undoes a transformation, though it only exists when the matrix is square and its determinant is not zero. Three operations that hold up the maths of a network.
Matrix multiplication is the core operation of a neural network: each layer gathers its weights into a matrix W and computes its output as the product W times X. That single operation, repeated layer after layer, turns the inputs into predictions and explains why graphics cards dominate modern deep learning.
A scalar is a single number, a vector an ordered list of numbers, a matrix a two-dimensional table and a tensor the generalisation to any number of dimensions. In a neural network the data enters as vectors and the weights form matrices, so every layer computes z = Wx + b by combining the two.
