Learning path Advanced
Training: Gradient Descent and Backpropagation
The learning engine: gradient descent, learning rate, SGD and mini-batch, step-by-step backpropagation, the computational graph, a layer's Jacobians and the loss landscape.
- 8 resources
- 4 views
- ~54 min
This path explains how neural networks actually learn: gradient descent and the algorithm that makes it work at scale, backpropagation, developed with the full math worked out step by step. It is built for readers who already understand a network’s forward pass (layers, weights, activation functions) and want to understand the learning mechanism in depth.
By the end you will be able to derive the gradient of a dense layer by hand, tell apart batch, stochastic and mini-batch gradient descent and when to use each, and reason about why a network converges, stalls at a local minimum, or gets stuck at a saddle point. This is advanced material: it assumes comfort with partial derivatives, matrix notation and the basic architecture of a feedforward network.
The sequence opens with the intuition behind gradient descent and the role of the learning rate, then compares the batch, stochastic and mini-batch variants used in practice. From there it moves into backpropagation: first the intuition for why it works, then its full mathematical derivation, and the computational graph that implements it through automatic differentiation. It closes with the gradients and Jacobians of a dense layer and an analysis of the loss landscape, local minima included, to explain why training behaves the way it does. A dense layer with just 100 neurons already produces thousands of parameters whose gradient must be recomputed at every step; backpropagation solves that in a single backward pass thanks to the chain rule.
Gradient Descent
Gradient descent is the algorithm that trains almost every neural network. It computes the slope of the loss function with respect to each weight and takes a small step in the opposite direction, controlled by the learning rate, until it reaches a minimum where the error stops falling and the model has converged.
The Learning Rate in Training
The learning rate is the hyperparameter that sets the size of each step when adjusting the weights during training. Too high a value makes the loss diverge; too low a value makes learning painfully slow. Typical values range from 0.001 with Adam to 0.1 with classic gradient descent.
Batch, Stochastic and Mini-Batch Gradient Descent
Batch gradient descent uses all the data at each step, the stochastic version uses a single sample and mini-batch picks an intermediate group, usually 32 to 256 examples. This guide compares the three variants, their cost, their noise and why mini-batch has become the standard for training modern networks.
Backpropagation: The Intuition
Backpropagation shares out the blame for the error among all the weights of a neural network. It propagates an error signal backwards layer by layer, multiplying by the local derivatives, and so obtains the gradient of every weight in a single pass. That idea, published in 1986, is what makes training deep networks possible.
Backpropagation: The Step-by-Step Derivation
Backpropagation applies the chain rule to share a network's error layer by layer. You first compute the error of the output layer, propagate it backward, and use it to obtain the gradients of every weight and bias in a single pass, summarised in four compact equations you can code directly.
The Computational Graph and Automatic Differentiation
A computational graph represents a function as a network of elementary operations. Automatic differentiation traverses that graph to compute exact derivatives: reverse mode, the basis of backpropagation, obtains the gradient of every weight in a single backward pass. It is the mechanism that makes training networks with billions of parameters possible.
Gradients of a Dense Layer: Matrices and Jacobians
In a dense layer with z equal to Wx plus b, the Jacobian matrix of the output with respect to the input is the weight matrix W itself. From it come the two training rules: the gradient with respect to the weights is delta times x transposed, and the gradient with respect to the input is W transposed times delta.
Local Minima, Saddle Points and the Loss Landscape
In large neural networks the loss surface almost never traps training in a bad local minimum. The real obstacle is saddle points, far more common in high dimensions. This article explains local versus global minima, why stochastic gradient descent slips past them, and what role convexity plays in the whole picture of optimisation.