Tag
Adam combines a first-order moment (a moving average of the gradient) and a second-order moment (an average of its squares) to give each parameter its own learning rate. With bias correction and the defaults β1=0.9, β2=0.999 and ε=1e-8, it converges fast with almost no tuning.
AdaGrad and RMSProp are optimisers that give every weight its own learning rate. AdaGrad accumulates the square of all past gradients and divides the step by its root, which slowly fades it out. RMSProp fixes this with a moving average that forgets the past using a decay factor of about 0.9.
Nesterov accelerated gradient improves classic momentum by evaluating the gradient at a look-ahead point, the one inertia is about to reach, instead of the current point. That anticipation step corrects the trajectory before overshooting and reaches O(1/k²) convergence on smooth convex problems, the fastest a first-order method can achieve.
Momentum is an improvement to gradient descent that accumulates a velocity from past gradients instead of looking only at the current one. With a coefficient β around 0.9, that inertia smooths the zigzag of plain SGD, pushes through long narrow valleys and makes training converge noticeably faster than gradient descent alone.
In large neural networks the loss surface almost never traps training in a bad local minimum. The real obstacle is saddle points, far more common in high dimensions. This article explains local versus global minima, why stochastic gradient descent slips past them, and what role convexity plays in the whole picture of optimisation.
Batch gradient descent uses all the data at each step, the stochastic version uses a single sample and mini-batch picks an intermediate group, usually 32 to 256 examples. This guide compares the three variants, their cost, their noise and why mini-batch has become the standard for training modern networks.
The learning rate is the hyperparameter that sets the size of each step when adjusting the weights during training. Too high a value makes the loss diverge; too low a value makes learning painfully slow. Typical values range from 0.001 with Adam to 0.1 with classic gradient descent.
Gradient descent is the algorithm that trains almost every neural network. It computes the slope of the loss function with respect to each weight and takes a small step in the opposite direction, controlled by the learning rate, until it reaches a minimum where the error stops falling and the model has converged.
Mean squared error (MSE) is the most common loss function in regression: it averages the square of the gap between each true value and its prediction. Squaring punishes big mistakes hard, keeps the function differentiable and gives gradient descent a clear, smooth target to minimise while training a model.
The exploding gradient problem happens when the gradient norm grows without control during backpropagation, especially in deep and recurrent networks with large weights. Training destabilises and the loss turns into NaN. Gradient clipping, together with good initialisation and normalisation, is the standard fix used today.
The vanishing gradient problem appears when the error signal shrinks as it backpropagates through many layers. The sigmoid and hyperbolic tangent functions have small derivatives, and their product tends toward zero, so the early layers barely learn. ReLU, careful initialisation and normalisation solve the problem.
A partial derivative measures how a function changes when you move just one of its variables and hold the rest fixed. The gradient gathers all those partial derivatives into a vector that points toward the steepest ascent; in a neural network, moving the opposite way lowers the error and drives the training.
A derivative measures the rate of change of a function: how much its output varies when the input changes a little. In a neural network, that slope tells us in which direction and how strongly to adjust each weight to reduce the error, and it is the foundation of gradient descent and backpropagation.
