The exploding gradient problem appears when the loss gradient grows without control during backpropagation and makes training blow up. Instead of taking small steps, the network applies huge corrections to its weights, the loss oscillates or turns into NaN, and the model stops learning. It is the mirror image of the vanishing gradient problem and mostly affects deep and recurrent networks. The good news: it is controlled with simple, well-tested techniques. The same explanation is available in Spanish.

Key formula $\mathbf{W}\leftarrow\mathbf{W}-\eta\nabla L,\quad \|\nabla L\|\to\infty$

Key takeaways

  • The exploding gradient problem occurs when the gradient norm $|\nabla L|$ grows exponentially layer by layer until it overflows the numeric range.
  • The main cause is large weights combined with many layers: the error is multiplied repeatedly during backpropagation.
  • Typical symptoms are a loss that jumps to NaN or inf, huge weight values and training that never converges.
  • Gradient clipping is the most direct fix: it caps the gradient norm at a threshold, usually between 1 and 5.
  • Good weight initialisation and normalisation layers prevent the problem from the network design onwards.

What is the exploding gradient problem?

Training a neural network means adjusting its weights with gradient descent: you compute the gradient $\nabla L$ of the loss with respect to each weight and take a step proportional to the learning rate $\eta$. The exploding gradient problem appears when that gradient, instead of staying in a reasonable range, grows uncontrollably as it propagates backward through the network.

Formally, a weight update is $\mathbf{W}\leftarrow\mathbf{W}-\eta\nabla L$. If $|\nabla L|$ reaches values of the order of $10^{10}$ or even $10^{30}$, the step stops being a fine correction and becomes a jump that throws the weight into a useless region of the space. When that number exceeds the maximum representable 32-bit floating point value (about $3.4\cdot10^{38}$), inf appears, and any later operation produces NaN. From there the network can no longer recover.

This phenomenon is the reverse of the vanishing gradient problem: in one case the gradient tends to zero and the network fails to learn, in the other it tends to infinity and the network learns too much and badly. Both come from the same repeated multiplication during backpropagation, described in the Wikipedia article on the vanishing gradient problem[1].

Causes: large weights and deep networks

Backpropagation computes the gradient by applying the chain rule across all layers. At each layer $^{(l)}$ two factors intervene: the derivative of the activation $f'(z)$ and the weight matrix $\mathbf{W}$. If the eigenvalues of those matrices are greater than 1, the gradient is multiplied by a factor above one at every layer it crosses backward.

The effect is exponential. With a growth factor of $\lambda=1.5$ per layer, an $L=20$-layer network multiplies the gradient by $\lambda^{L}=1.5^{20}$, roughly 3,325 times; with 50 layers the factor exceeds 600 million. That is why the problem worsens with depth and hits recurrent networks especially hard, where the same weight matrix is applied once per time step of the sequence.

Show the derivation

By the chain rule, the gradient reaching layer $l$ is the product of the factors of every later layer: $\nabla L^{(l)}=\left(\prod_{k=l}^{L}\mathbf{W}_k^{\top}\operatorname{diag}(f'(z_k))\right)\nabla L^{(L)}$. If each factor multiplies the norm by a dominant eigenvalue $\lambda$, then $\|\nabla L^{(l)}\|\approx\lambda^{L-l}\,\|\nabla L^{(L)}\|$. With $\lambda>1$ that term grows exponentially as you backpropagate (explosion); with $\lambda<1$ it tends to zero (vanishing). The threshold sits exactly at $\lambda=1$.

The three most common concrete causes are: weight initialisation with values that are too large, an excessive learning rate $\eta$, and very deep architectures without stabilisation mechanisms. The reference paper on this topic, «On the difficulty of training recurrent neural networks»[2] by Pascanu, Mikolov and Bengio (2013), shows that in recurrent networks it is enough for a dominant eigenvalue to exceed 1 for the gradient to explode over time.

Gradient clipping

The most direct and widely used solution is gradient clipping. The idea is geometric and elegant: when the gradient explodes, its direction is usually still correct, what fails is its length. So there is no need to change the direction, only to shorten the vector.

The recommended method is clipping by norm. You set a threshold $c$ (a typical value between 1 and 5) and, if the gradient norm exceeds it, you rescale the whole vector so its norm is exactly $c$, preserving the direction:

$$\text{if } |\nabla L|>c:\quad \nabla L\leftarrow c\cdot\frac{\nabla L}{|\nabla L|}$$

Clipping by norm does not change the gradient’s direction, only its length: dividing by $\|\nabla L\|$ and multiplying by $c$ yields a vector that points the same way but with norm exactly $c$. That is why it preserves the information about which way to descend.

As Pascanu, Mikolov and Bengio put it, «we propose a gradient norm clipping strategy to deal with exploding gradients». There is also clipping by value, which limits each component separately to a range [-c, c], but it has weaker theoretical justification because it alters the gradient’s direction. In practice, frameworks such as PyTorch and TensorFlow offer both variants in a single line of code, and clipping by norm is the default in most language-model training recipes.

Initialisation and normalisation as prevention

Clipping treats the symptom; initialisation and normalisation attack the cause. Starting training with weights of the right size stops the gradient from growing in the first place. The Xavier (Glorot) and He strategies scale the initial variance of the weights according to the number of inputs and outputs of each layer, so the signal neither dies out nor blows up as it crosses the network.

Batch normalisation, introduced in 2015, rescales the activations of each layer to have mean 0 and variance 1, which keeps gradients in a stable range and allows training networks with hundreds of layers. ResNet’s residual connections, which in 2015 made 152-layer networks viable, play a similar role by creating shortcuts through which the gradient flows without being multiplied. The Deep Learning[3] textbook by Goodfellow, Bengio and Courville devotes its chapter 8 to these optimisation strategies. Combining good initialisation, normalisation and gradient clipping is today the standard recipe for training deep networks smoothly.

Example

Imagine a recurrent network processing a 100-step sequence whose weight matrix has a dominant eigenvalue of $1.2$. When backpropagating the error from the last step to the first, the gradient is multiplied by roughly $1.2^{100}$, a number close to $8.3\cdot10^{7}$. A gradient that should be $0.5$ reaches the first step turned into more than 40 million.

With a learning rate $\eta$ of $0.01$, the weight update would be about 400,000 in a single step: the weight jumps from a healthy value near 0 to a monstrous figure, the activation $a=f(z)$ saturates, and on the next iteration the loss $L$ becomes NaN. If we apply gradient clipping with threshold $c=1$, that 40-million gradient is rescaled to norm 1 before the update, the step is a fine correction again, and training continues stably. This contrast, measured in any practice notebook, is the best way to internalise why clipping is so effective. You can review the mechanics of the update and place this piece within the mathematics behind neural networks roadmap.

Frequently asked questions

How do I know if my network suffers from exploding gradients?

The signs are clear: the loss suddenly jumps to NaN or inf, weight values grow out of control between iterations, and training curves show sharp spikes instead of a smooth decline. Logging the gradient norm at each step confirms the diagnosis: if you see values spanning several orders of magnitude, you are facing an exploding gradient.

Is the exploding gradient the same as the vanishing gradient?

No, they are the two extremes of the same mechanism. In the vanishing gradient the norm tends to zero and the early layers stop learning; in the exploding one the norm tends to infinity and training destabilises. Both arise from multiplying many factors during backpropagation, but they are tackled with different techniques.

What clipping threshold should I use?

There is no universal value, but thresholds between 1 and 5 work well in most cases and are the most common when training language models. Start with a value like 1, watch the gradient norm during the first iterations, and adjust if training stays unstable or, conversely, progresses too slowly.

Conclusion

The exploding gradient problem is no mystery but the mathematical consequence of multiplying factors greater than 1 across a deep network. Its symptoms (loss at NaN, monstrous weights, unstable training) are easy to recognise, and its remedies are well established: gradient clipping for the immediate symptom, and careful initialisation and normalisation to prevent it by design. With these tools, training networks with tens or hundreds of layers stops being a lottery. The natural next step is to understand its twin, the vanishing gradient problem.

Sources

  1. Wikipedia article on the vanishing gradient problem
  2. «On the difficulty of training recurrent neural networks»
  3. Deep Learning
  4. On the difficulty of training Recurrent Neural Networks (arXiv:1211.5063)

Route: The Neuron and Activation Functions