Nesterov Accelerated Gradient (NAG)

Nesterov accelerated gradient improves classic momentum by evaluating the gradient at a look-ahead point, the one inertia is about to reach, instead of the current point. That anticipation step corrects the trajectory before overshooting and reaches O(1/k²) convergence on smooth convex problems, the fastest a first-order method can achieve.

July 16, 2026 6 min

Technology

Momentum in Gradient Descent

Momentum is an improvement to gradient descent that accumulates a velocity from past gradients instead of looking only at the current one. With a coefficient β around 0.9, that inertia smooths the zigzag of plain SGD, pushes through long narrow valleys and makes training converge noticeably faster than gradient descent alone.

July 16, 2026 8 min