Tag
A computational graph represents a function as a network of elementary operations. Automatic differentiation traverses that graph to compute exact derivatives: reverse mode, the basis of backpropagation, obtains the gradient of every weight in a single backward pass. It is the mechanism that makes training networks with billions of parameters possible.
Backpropagation applies the chain rule to share a network's error layer by layer. You first compute the error of the output layer, propagate it backward, and use it to obtain the gradients of every weight and bias in a single pass, summarised in four compact equations you can code directly.
Backpropagation shares out the blame for the error among all the weights of a neural network. It propagates an error signal backwards layer by layer, multiplying by the local derivatives, and so obtains the gradient of every weight in a single pass. That idea, published in 1986, is what makes training deep networks possible.
The exploding gradient problem happens when the gradient norm grows without control during backpropagation, especially in deep and recurrent networks with large weights. Training destabilises and the loss turns into NaN. Gradient clipping, together with good initialisation and normalisation, is the standard fix used today.
The chain rule computes the derivative of a composite function by multiplying the derivatives of its links: if y depends on u and u depends on x, then dy/dx equals dy/du times du/dx. That layer-by-layer multiplication of derivatives is exactly what backpropagation does to train a neural network.
A multilayer neural network consists of an input layer, one or more hidden layers, and an output layer, where each neuron weights its inputs and applies a non-linear activation function before passing the result to the next layer. Through forward propagation and backpropagation, the network adjusts millions of weights to learn hierarchical representations capable of classifying images, translating text, or generating language.
