Momentum in Gradient Descent
About this video
How momentum builds velocity and smooths gradient descent.
Momentum is an improvement to gradient descent that accumulates a velocity from past gradients instead of looking only at the current one. With a coefficient β around 0.9, that inertia smooths the zigzag of plain SGD, pushes through long narrow valleys and makes training converge noticeably faster than gradient descent alone.
Video transcript
- 0:01 With momentum, each step keeps part of the previous step's velocity.
- 0:05 Compared with plain gradient descent, the ball builds up speed and reaches the
- 0:10 minimum sooner.
More videos
Explainer
Firecrawl: Web Data for Agents
If you write your own crawler: robots.txt, one queue per host and 304 Not Modified make the polite option the cheap one too.
Explainer
How to Install SearXNG with Docker
One use for your own instance: an AI agent that finds its own sources by querying SearXNG, without paying per search.
Explainer
The Computational Graph and Automatic Differentiation
The computational graph and automatic differentiation, in action.
Explainer
Backpropagation: The Step-by-Step Derivation
The backpropagation derivation, factor by factor with the chain rule.
Explainer
Backpropagation: The Intuition
The intuition of backpropagation: the error flows backward through the network.