Adam is the optimizer that most deep learning projects reach for by default, and for good reason: it usually just works with barely any tuning. Published by Diederik Kingma and Jimmy Ba in 2014, it merges two earlier ideas (momentum and adaptive rates) into a single update rule that adapts to each parameter. Its cousin AdamW fixes a subtle regularization detail that mattered more than it first appeared. The same explanation is available in Spanish.

Key formula $w_t = w_{t-1}-\eta\,\dfrac{\hat{m}_t}{\sqrt{\hat{v}_t}+\varepsilon}$

Key takeaways

  • The Adam optimizer keeps two moving averages per parameter: the first-order moment (the gradient) and the second-order moment (its square).
  • The defaults are β1=0.9, β2=0.999 and ε=1e-8, with a typical learning rate of 0.001; you rarely need to change them.
  • Bias correction compensates for both averages starting at zero, avoiding tiny steps in the first iterations.
  • AdamW separates weight decay from the adaptive gradient, something the original Adam did poorly and which weakened regularization.
  • Against SGD with momentum, Adam converges faster early on, though a well-tuned SGD sometimes generalizes slightly better in computer vision.

What is Adam?

Adam (Adaptive Moment Estimation) is a gradient descent method that tunes the learning rate of each weight separately. Instead of moving every parameter with the same step, it watches each one’s recent gradient history and decides how far to advance.

To do so it keeps two accumulators per parameter. The first is an average of the gradient (which way the error points); the second is an average of the squared gradient (how large and noisy it is). Dividing the first by the square root of the second yields a normalized step: large when the gradient is stable and small when it oscillates. That machinery inherits the spirit of AdaGrad and RMSprop but adds momentum to smooth the direction.

The full rule, at time step $t$, reads:

m = beta1 * m + (1 - beta1) * g        # first moment: average of the gradient
v = beta2 * v + (1 - beta2) * g**2     # second moment: average of the square
m_hat = m / (1 - beta1**t)             # bias correction of the first moment
v_hat = v / (1 - beta2**t)             # bias correction of the second moment
w = w - lr * m_hat / (v_hat**0.5 + eps)

The term $\varepsilon$ (by default $10^{-8}$) only prevents division by zero when $\hat{v}_t$ is tiny; it is not a hyperparameter worth tuning.

First- and second-order moments

The method’s name comes from those two "moments", a term borrowed from statistics. The first-order moment $m$ is an exponential average of the gradient $g$: it accumulates the recent direction and filters the noise, just like classic momentum. With β1=0.9, each new iteration weighs 10% and carries 90% of the previous history.

$$m_t = \beta1\,m{t-1}+(1-\beta_1)\,g_t \qquad v_t = \beta2\,v{t-1}+(1-\beta_2)\,g_t^2$$

The second-order moment $v$ is an exponential average of the squared gradient, with β2=0.999. It estimates each parameter’s recent variance. When a weight receives large, erratic gradients, its $v$ grows and the effective step shrinks; when the gradient is small and steady, the step grows. Each dimension gets its own learning rate without our fixing it by hand.

Bias correction

Here is the detail that sets Adam apart from a plain mix of momentum and RMSprop. Because $m$ and $v$ start at zero, during the first iterations both averages are biased downward: they underestimate the real value. Left uncorrected, the optimizer would take overly timid steps right at the start of training, exactly when advancing matters most.

The fix is to divide each average by $(1-\beta^t)$, a factor close to zero at first (so it amplifies the estimate) that tends to 1 as $t$ grows:

$$\hat{m}_t = \frac{m_t}{1-\beta_1^{\,t}} \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^{\,t}}$$

With β2=0.999 that factor only nears 1 after a few hundred steps, so the correction matters most for the second moment. It is a cheap adjustment that stabilizes the early epochs at no noticeable cost.

Show the derivation

Unrolling the recurrence $m_t = \beta_1 m_{t-1}+(1-\beta_1)g_t$ from $m_0 = 0$ gives $m_t = (1-\beta_1)\sum_{i=1}^{t}\beta_1^{\,t-i}\,g_i$. If the gradient is roughly stationary with mean $\mathbb{E}[g]$, then $\mathbb{E}[m_t] = (1-\beta_1^{\,t})\,\mathbb{E}[g]$. Dividing by $(1-\beta_1^{\,t})$ leaves $\mathbb{E}[\hat{m}_t] = \mathbb{E}[g]$, an unbiased estimator; the same argument holds for $\hat{v}_t$.

AdamW and decoupled weight decay

In 2017 Ilya Loshchilov and Frank Hutter pointed out a subtle mistake in how Adam applied regularization. In SGD, adding a weight decay term to the gradient is exactly equivalent to L2 regularization. But not in Adam: passing through the division by the square root of the second moment scales that term differently for each parameter, and regularization loses strength precisely where the gradient is large.

Their proposal, AdamW, decouples weight decay: rather than folding it into the gradient, it is subtracted directly from the weights, apart from the adaptive step:

$$wt = w{t-1}-\eta\,\frac{\hat{m}_t}{\sqrt{\hat{v}t}+\varepsilon}-\eta\,\lambda\,w{t-1}$$

As the authors put it, «L2 regularization and weight decay are not identical for Adam». The change is small in code but improves generalization measurably, and today AdamW is the default in most libraries and in the training of large language models.

Adam versus SGD with momentum

Adam usually wins on early speed: it reaches a low loss in fewer iterations and is more forgiving of a poor learning-rate choice. That makes it the recommended starting point when you explore a new architecture or train transformers.

SGD with momentum, by contrast, has a reputation for generalizing slightly better on some computer vision tasks if it is carefully tuned and paired with a good learning-rate schedule. The gap has narrowed with AdamW, but the choice still depends on the problem. A useful guide for comparing optimizer families is Sebastian Ruder’s overview of gradient descent. For the full mathematical context, the mathematics of neural networks roadmap places every piece where it belongs.

Frequently asked questions

What defaults does Adam use?

The standard parameters are β1=0.9, β2=0.999 and ε=1e-8, with a common initial learning rate of 0.001. They are the values the original paper proposed and they work well in the vast majority of cases without tuning.

What is the difference between Adam and AdamW?

AdamW changes only how weight decay is applied: it subtracts it directly from the weights instead of adding it to the gradient. That decoupling restores the regularization effect the original Adam diluted, and it usually improves generalization with no other changes.

Should I always use Adam?

It is an excellent default, especially as AdamW. Even so, for some vision tasks a well-tuned SGD with momentum can generalize slightly better, so it is worth trying both when final performance is the priority.

Conclusion

Adam works because it combines three simple ideas: momentum smooths the direction, the second moment adapts the step to each parameter, and bias correction stabilizes the start. AdamW adds the missing piece by separating weight decay from the adaptive step. With its defaults (β1=0.9, β2=0.999, ε=1e-8) you get a robust optimizer for almost any network. The natural next step is to understand how the learning rate shapes convergence speed.

Sources

  1. Adam: A Method for Stochastic Optimization (Kingma and Ba, 2014)
  2. Decoupled Weight Decay Regularization (Loshchilov and Hutter, 2017)
  3. An overview of gradient descent optimization algorithms (Sebastian Ruder)
  4. torch.optim.Adam (PyTorch documentation)

Route: Advanced Neural Network Optimization