Advanced Neural Network Optimization
Beyond basic gradient descent: momentum, Nesterov, AdaGrad and RMSProp, Adam and AdamW, learning rate scheduling and second-order methods.
-
Momentum in Gradient Descent
Momentum is an improvement to gradient descent that accumulates a velocity from past gradients instead of looking only at the current one. With a coefficient β around 0.9, that inertia smooths the zigzag of plain SGD, pushes through long narrow valleys and makes training converge noticeably faster than gradient descent alone.
-
Nesterov Accelerated Gradient (NAG)
Nesterov accelerated gradient improves classic momentum by evaluating the gradient at a look-ahead point, the one inertia is about to reach, instead of the current point. That anticipation step corrects the trajectory before overshooting and reaches O(1/k²) convergence on smooth convex problems, the fastest a first-order method can achieve.
-
AdaGrad and RMSProp: Adaptive Learning Rates
AdaGrad and RMSProp are optimisers that give every weight its own learning rate. AdaGrad accumulates the square of all past gradients and divides the step by its root, which slowly fades it out. RMSProp fixes this with a moving average that forgets the past using a decay factor of about 0.9.
-
Adam and AdamW: The Default Optimizer
Adam combines a first-order moment (a moving average of the gradient) and a second-order moment (an average of its squares) to give each parameter its own learning rate. With bias correction and the defaults β1=0.9, β2=0.999 and ε=1e-8, it converges fast with almost no tuning.
-
Learning Rate Schedules and Warmup
A learning rate schedule changes the value of η over the course of training instead of keeping it fixed. It starts with a warmup that raises η from near zero, holds a peak and then lowers it with step, exponential or cosine decay so the network converges faster and with far less oscillation.
-
Second-Order Methods: Newton and Its Approximations
Second-order methods use the Hessian, the matrix of second derivatives of the loss, to orient each step better than the gradient alone. Newton's method updates the weights by subtracting the inverse Hessian times the gradient, but inverting it is far too costly in large networks, so practitioners fall back on quasi-Newton approximations.