Training: Gradient Descent and Backpropagation
The learning engine: gradient descent, learning rate, SGD and mini-batch, step-by-step backpropagation, the computational graph, a layer's Jacobians and the loss landscape.
-
Gradient Descent
Gradient descent is the algorithm that trains almost every neural network. It computes the slope of the loss function with respect to each weight and takes a small step in the opposite direction, controlled by the learning rate, until it reaches a minimum where the error stops falling and the model has converged.
-
The Learning Rate in Training
The learning rate is the hyperparameter that sets the size of each step when adjusting the weights during training. Too high a value makes the loss diverge; too low a value makes learning painfully slow. Typical values range from 0.001 with Adam to 0.1 with classic gradient descent.
-
Batch, Stochastic and Mini-Batch Gradient Descent
Batch gradient descent uses all the data at each step, the stochastic version uses a single sample and mini-batch picks an intermediate group, usually 32 to 256 examples. This guide compares the three variants, their cost, their noise and why mini-batch has become the standard for training modern networks.
-
Backpropagation: The Intuition
Backpropagation shares out the blame for the error among all the weights of a neural network. It propagates an error signal backwards layer by layer, multiplying by the local derivatives, and so obtains the gradient of every weight in a single pass. That idea, published in 1986, is what makes training deep networks possible.
-
Backpropagation: The Step-by-Step Derivation
Backpropagation applies the chain rule to share a network's error layer by layer. You first compute the error of the output layer, propagate it backward, and use it to obtain the gradients of every weight and bias in a single pass, summarised in four compact equations you can code directly.
-
The Computational Graph and Automatic Differentiation
A computational graph represents a function as a network of elementary operations. Automatic differentiation traverses that graph to compute exact derivatives: reverse mode, the basis of backpropagation, obtains the gradient of every weight in a single backward pass. It is the mechanism that makes training networks with billions of parameters possible.
-
Gradients of a Dense Layer: Matrices and Jacobians
In a dense layer with z equal to Wx plus b, the Jacobian matrix of the output with respect to the input is the weight matrix W itself. From it come the two training rules: the gradient with respect to the weights is delta times x transposed, and the gradient with respect to the input is W transposed times delta.
-
Local Minima, Saddle Points and the Loss Landscape
In large neural networks the loss surface almost never traps training in a bad local minimum. The real obstacle is saddle points, far more common in high dimensions. This article explains local versus global minima, why stochastic gradient descent slips past them, and what role convexity plays in the whole picture of optimisation.