The learning rate decides how far a neural network moves at each step of training. It is a single number, almost always between 0.001 and 0.1, and it governs whether the model converges, stalls or diverges beyond repair. Tuning it well usually matters more than any other hyperparameter. This guide explains what it is, how it affects learning and how to pick a sensible value. It is part of the mathematics behind neural networks roadmap, and the same explanation is available in Spanish.

Key formula $\mathbf{w} \leftarrow \mathbf{w}-\eta\cdot\nabla L$

Key takeaways

  • The learning rate ($\eta$) sets the step size in the update rule $\mathbf{w} \leftarrow \mathbf{w}-\eta\cdot\nabla L$.
  • Too high a value makes the loss oscillate or diverge; too low a value stretches training across thousands of iterations.
  • For the Adam optimiser, 0.001 is the usual starting point; for stochastic gradient descent, between 0.1 and 0.01.
  • The practical way to find it is to sweep on a log scale (0.1, 0.01, 0.001) and compare the loss curve.
  • Schedules adjust $\eta$ over the course of training: they start high to move fast and drop to fine-tune the result.

What is the learning rate?

Training a network means repeating a very simple move: measure the error, work out which way it grows with the gradient and take a step in the opposite direction. The learning rate is the number that scales that step. In the update rule $\mathbf{w} \leftarrow \mathbf{w}-\eta\cdot\nabla L$, the Greek letter $\eta$ (eta) is that rate: the larger it is, the longer the jump each weight $\mathbf{w}$ takes against the gradient $\nabla L$.

Think of it as the stride length while walking down a foggy mountain. The gradient tells you the slope under your feet; the learning rate decides whether you move in short, safe steps or in large, risky leaps. It is a hyperparameter, a value you fix before training that the model does not learn on its own, unlike the weights. The full mechanism is covered in gradient descent.

Too high versus too low

The balance of $\eta$ is clearest at its two extremes. With a rate that is too high, each step overshoots the point you were aiming for: the loss bounces from one side of the valley to the other and can even blow up (a NaN appears in the log). That is divergence. With a rate that is too low the opposite happens: the model does move, but so slowly that you would need 10 or 100 times more iterations to reach the same place, and it often gets stuck on a flat region too early.

Between those two failures lies a wide band of values that work. So the goal is not to hit the perfect number but to land inside that band. A clear symptom of an excessive rate is a loss curve that rises in the first iterations instead of falling; a symptom of too small a rate is an almost flat curve that barely improves after hundreds of steps.

How to choose a good value

The most reliable method is a sweep on a log scale: try 0.1, 0.01 and 0.001, watch each one’s loss curve for a few epochs and keep the largest value that still falls steadily. There is an automated variant, the learning rate range test, which raises $\eta$ gradually over one epoch and locates the point where the loss starts to shoot up. Stanford’s CS231n[1] notes collect these heuristics with example curves.

As reasonable starting points, Adam works well around 0.001 (and the famous 3e-4, that is 0.0003, has become almost a cliché among practitioners), while stochastic gradient descent with momentum usually starts at 0.1. The standard textbook puts it bluntly: «the learning rate is perhaps the most important hyperparameter», in the words of Goodfellow, Bengio and Courville in Deep Learning[2]. If you only have time to tune one value, tune this one.

Relationship with convergence

Convergence is the process by which the loss settles near a minimum. The learning rate controls both its speed and its stability at once, and that is the tension: raising it speeds up progress but risks stability, and lowering it guarantees stability but slows everything down. In simple convex problems, theory fixes a maximum rate above which the method stops converging; in deep networks that boundary has no closed formula and is found empirically.

For a convex function with an $L$-Lipschitz gradient, gradient descent converges only if $\eta<2/L$; above that threshold each step amplifies the error instead of shrinking it. A deep network has no such closed-form bound, which is why the rate is found with an empirical sweep.

A common technique to avoid choosing between the two virtues is warmup: starting with a very small $\eta$ for the first few hundred steps and raising it gradually, so the network does not destabilise while the weights are still freshly initialised. Sebastian Ruder keeps a widely cited synthesis of these ideas in his overview of optimisation methods[3].

A preview of schedules

Keeping $\eta$ constant throughout training is rarely optimal. Schedules change its value over time: high at first to cover a lot of ground and low at the end to settle into the minimum. The most common are step decay (dividing $\eta$ by 10 every so many epochs), cosine decay and the warmup already mentioned, which is often combined with the others.

The intuition is the same mountain stride: you take long steps while far from the goal and shorten them as you get closer. This dynamic adjustment is the subject of the next article in the series; here it is enough to know that the learning rate need not be a fixed number. You can widen the context in the Wikipedia entry on the learning rate[4].

Frequently asked questions

What is a good learning rate to start with?

It depends on the optimiser. With Adam, 0.001 is a solid starting point for most tasks; with stochastic gradient descent and momentum, try 0.1 or 0.01. In any case, run a quick sweep on a log scale before launching a long training run.

What if the loss goes up instead of down?

It almost always means the learning rate is too high and each step overshoots the minimum. Divide it by 3 or by 10 and try again. If the loss turns into NaN, that is the clearest sign of divergence caused by an excessive $\eta$.

Is the learning rate the same as the batch size?

No. The batch size is how many examples you use to estimate each gradient, while the learning rate is how far you move with that estimate. They are related (larger batches tolerate somewhat larger rates), but they are two distinct hyperparameters.

Conclusion

The learning rate is the dial that governs the speed and safety of training with a single number. Too high and the model diverges; too low and it makes no progress; at the right value, the loss falls fast and steadily. Start with 0.001 for Adam or 0.1 for stochastic gradient descent, run a log-scale sweep and, once you have mastered the fixed value, take the step toward the schedules that make it vary during training.

Sources

  1. CS231n
  2. Deep Learning
  3. his overview of optimisation methods
  4. the Wikipedia entry on the learning rate

Route: Training: Gradient Descent and Backpropagation