Local Minima, Saddle Points and the Loss Landscape
Table of contents
- Key takeaways
- The loss surface in high dimensions
- Local minima versus global minima
- Saddle points, the real obstacle
- Why SGD escapes them
- Convexity
- Frequently asked questions
- Are local minima a real problem when training networks?
- What is the difference between a local minimum and a saddle point?
- Why does stochastic gradient descent avoid saddle points?
- Conclusion
- Sources
For years the great enemy of training a network was thought to be getting stuck in a local minimum; in high dimensions the reality is different. In a space of millions of parameters, bad local minima are extremely rare, and the real brake is saddle points: flat regions where the gradient nearly vanishes yet that are not the bottom of the valley. Understanding the shape of the loss surface explains why training works so well despite solving a non-convex problem. The same explanation is available in Spanish.
Key takeaways
- The loss surface is the function that assigns an error to every combination of weights; training is descending it until a critical point where the gradient $\nabla L = \mathbf{0}$.
- Local minima are points where no direction reduces the error, but in large networks almost all of them sit very close in value to the global minimum.
- Saddle points, not local minima, are the dominant obstacle: in 2014 Dauphin and colleagues showed they proliferate exponentially with dimension.
- Stochastic gradient descent escapes saddle points because its noise pushes it out of the descending directions that the exact gradient cannot see.
- A convex problem has a single minimum; training a network is non-convex, and even so the result is surprisingly good.
The loss surface in high dimensions
Picture the loss function as a terrain of mountains and valleys: each position represents a set of weights and the height is the error the network makes. With two weights that terrain fits in a three-dimensional plot, but a modern network has millions or billions of parameters, so its loss surface lives in a space of that many dimensions. That change of scale changes everything.
In two or three dimensions our intuition says a closed valley (a local minimum) is easy to find and hard to leave. In a million dimensions that intuition fails. For a point to be a local minimum, the error has to rise in all 1,000,000 directions at once; if it falls in even one, it is not. The chance that all 1,000,000 point upward is tiny, which is why true local minima are scarce. The reference book Deep Learning[1] devotes its chapter 8 to exactly this geometry of optimisation.
Local minima versus global minima
A global minimum is the point of lowest possible error across the whole surface. A local minimum is the bottom of any valley: it reduces the error relative to its immediate surroundings, but a deeper valley may exist elsewhere. For a long time people feared that training would get pinned in a poor local minimum, far from the global one.
A decade of research took that fear apart. The work by Choromanska and colleagues, The Loss Surfaces of Multilayer Networks[2] (2015), showed that in large networks the vast majority of local minima cluster in a narrow band of values very close to the global minimum. In other words: if gradient descent finds a local minimum, that minimum is almost certainly good enough. Finding the perfect optimum stops being the goal when any valley floor will do.
Saddle points, the real obstacle
A saddle point is a place where the gradient vanishes, just as at a minimum, but it is neither peak nor floor: in some directions the error rises and in others it falls, like the riding saddle it is named after. Because the gradient is zero, a pure gradient descent stalls there, fooled into thinking it has finished.
The condition $\nabla L = \mathbf{0}$ is satisfied by minima, maxima and saddles alike. What tells the type apart are the signs of the eigenvalues of the Hessian matrix $\nabla^2 L$: all positive means a minimum, all negative a maximum, and mixed signs a saddle point.
In high dimensions saddle points are not a curiosity but the norm. Random matrix theory predicts that, among all critical points with high error, the ratio of saddle points to minima grows explosively with dimension. The foundational paper by Yann Dauphin and colleagues, Identifying and attacking the saddle point problem[3] (2014), states it plainly: «a deeper and more profound difficulty originates from the proliferation of saddle points, not local minima». These points are surrounded by plateaus of nearly flat error that slow learning for many iterations.
Why SGD escapes them
This is where noise turns out to be a virtue. Exact gradient descent uses all the data to compute the descent direction, and at a saddle point that direction is zero. Stochastic gradient descent instead estimates the gradient from a small batch of examples, so every step carries a sampling error. That noise acts as a random nudge that is almost never exactly zero along the descending directions, and the smallest nudge is enough for the optimisation to roll off the saddle and keep falling.
That is why the batched, noisy variants we review in batch, stochastic and mini-batch gradient descent are not only faster per iteration: they also avoid getting stuck. Modern optimisers such as Adam add momentum and adaptive learning rates that speed up leaving those flat plateaus even more.
Convexity
A function is convex when the segment joining any two points of its graph always stays above the curve; in practice that means it has a single valley, with no saddle points or secondary local minima. In a convex problem, such as linear regression, gradient descent has a mathematical guarantee of reaching the one global minimum. It is the ideal world of optimisation, and you can review the formal definition on Wikipedia[4].
Training a neural network is not convex: its stacked layers and non-linear activations create a surface full of saddle points and local minima. The paradox is that, despite losing every theoretical guarantee, the method works beautifully in practice, precisely because of the geometry described above. To place this piece within the whole, the roadmap of the mathematics behind neural networks connects optimisation with linear algebra and calculus.
Frequently asked questions
Are local minima a real problem when training networks?
In practice, almost never. In large networks most local minima have an error very close to that of the global minimum, so landing in one rarely hurts the model. The obstacle that does matter is saddle points and their flat plateaus.
What is the difference between a local minimum and a saddle point?
In both the gradient is zero. At a local minimum the error rises in every direction; at a saddle point it rises in some and falls in others. That is why a local minimum is a stable stop and a saddle point is only a temporary trap that the noise of SGD helps you leave.
Why does stochastic gradient descent avoid saddle points?
Because it estimates the gradient from a small batch and makes a sampling error at every step. That noise breaks the tie that would keep an exact method motionless and pushes the optimisation toward the directions where the error can still fall.
Conclusion
The shape of the loss surface explains one of the big surprises of deep learning: training a non-convex problem in millions of dimensions works because local minima are almost all good and because the noise of stochastic descent dissolves saddle points. Local minima stopped being the villain; the real challenge is crossing the flat plateaus efficiently. The natural next step is to see how optimisers do it in gradient descent.
