Learning path Advanced
Advanced Neural Network Optimization
Beyond basic gradient descent: momentum, Nesterov, AdaGrad and RMSProp, Adam and AdamW, learning rate scheduling and second-order methods.
- 6 resources
- 6 views
- ~43 min
This learning path explains how modern deep learning optimizers work and why they matter, from classic momentum to second-order methods, for readers who already know basic gradient descent and want to understand what powers Adam, AdamW, or a learning rate warmup.
What you will be able to do
By the end of this path you will be able to explain and choose the right optimizer for a given problem: what actually speeds up training, when an adaptive learning rate helps, and how to diagnose training that oscillates or stalls. This is an advanced-level path: it assumes you already know gradient descent, backpropagation, and the basic matrix algebra behind a neural network.
How the sequence builds
It starts with momentum in gradient descent, the idea of accumulating velocity along the gradient direction to dampen zigzagging on elongated error surfaces. From there, Nesterov Accelerated Gradient (NAG) fixes a limitation of classic momentum by looking “one step ahead” before computing the gradient. The path continues with AdaGrad and RMSProp, which introduce per-parameter adaptive learning rates, and builds up to Adam and AdamW, deep learning’s de facto default optimizer because it combines momentum with adaptation. It closes with learning rate scheduling (schedules and warmup) and with second-order methods derived from Newton’s method, which use curvature instead of just slope.
Adam has been cited in more than 100,000 papers since its 2015 publication, a sign of how much this family of optimizers has shaped modern neural network training.
Momentum in Gradient Descent
Momentum is an improvement to gradient descent that accumulates a velocity from past gradients instead of looking only at the current one. With a coefficient β around 0.9, that inertia smooths the zigzag of plain SGD, pushes through long narrow valleys and makes training converge noticeably faster than gradient descent alone.
Nesterov Accelerated Gradient (NAG)
Nesterov accelerated gradient improves classic momentum by evaluating the gradient at a look-ahead point, the one inertia is about to reach, instead of the current point. That anticipation step corrects the trajectory before overshooting and reaches O(1/k²) convergence on smooth convex problems, the fastest a first-order method can achieve.
AdaGrad and RMSProp: Adaptive Learning Rates
AdaGrad and RMSProp are optimisers that give every weight its own learning rate. AdaGrad accumulates the square of all past gradients and divides the step by its root, which slowly fades it out. RMSProp fixes this with a moving average that forgets the past using a decay factor of about 0.9.
Adam and AdamW: The Default Optimizer
Adam combines a first-order moment (a moving average of the gradient) and a second-order moment (an average of its squares) to give each parameter its own learning rate. With bias correction and the defaults β1=0.9, β2=0.999 and ε=1e-8, it converges fast with almost no tuning.
Learning Rate Schedules and Warmup
A learning rate schedule changes the value of η over the course of training instead of keeping it fixed. It starts with a warmup that raises η from near zero, holds a peak and then lowers it with step, exponential or cosine decay so the network converges faster and with far less oscillation.
Second-Order Methods: Newton and Its Approximations
Second-order methods use the Hessian, the matrix of second derivatives of the loss, to orient each step better than the gradient alone. Newton's method updates the weights by subtracting the inverse Hessian times the gradient, but inverting it is far too costly in large networks, so practitioners fall back on quasi-Newton approximations.