Tag
In a dense layer with z equal to Wx plus b, the Jacobian matrix of the output with respect to the input is the weight matrix W itself. From it come the two training rules: the gradient with respect to the weights is delta times x transposed, and the gradient with respect to the input is W transposed times delta.
Forward propagation is the process by which a neural network turns its input into a prediction, one layer at a time. Each layer multiplies the input vector by a weight matrix, adds a bias and applies an activation function, so the output of one layer feeds the next until the final result appears at the end.
A vector norm measures its length or magnitude. The L1 norm adds the absolute values of the components, while the L2 norm applies the Pythagorean theorem: the square root of the sum of squares. Both define different distances between points and underpin the regularisation that prevents overfitting in neural networks.
The transpose swaps rows for columns and shows up at every step of backpropagation; the identity matrix acts as the 1 of matrix algebra and leaves any vector unchanged, and the inverse matrix undoes a transformation, though it only exists when the matrix is square and its determinant is not zero. Three operations that hold up the maths of a network.
Matrix multiplication is the core operation of a neural network: each layer gathers its weights into a matrix W and computes its output as the product W times X. That single operation, repeated layer after layer, turns the inputs into predictions and explains why graphics cards dominate modern deep learning.
The dot product multiplies each input by its weight and adds the results into a single number. A neuron uses that operation to compute its weighted sum z equals w times x plus the bias b, and that value decides, after the activation, how strongly the neuron fires in response to the data it receives.
A scalar is a single number, a vector an ordered list of numbers, a matrix a two-dimensional table and a tensor the generalisation to any number of dimensions. In a neural network the data enters as vectors and the weights form matrices, so every layer computes z = Wx + b by combining the two.
The mathematics of neural networks rests on three blocks: linear algebra represents data and weights as vectors and matrices, calculus with derivatives and the chain rule lets the network learn through gradient descent, and probability shapes the loss functions. This roadmap walks that path from beginning to end so you know what to study and in what order.
