The Computational Graph and Automatic Differentiation
About this video
The computational graph and automatic differentiation, in action.
A computational graph represents a function as a network of elementary operations. Automatic differentiation traverses that graph to compute exact derivatives: reverse mode, the basis of backpropagation, obtains the gradient of every weight in a single backward pass. It is the mechanism that makes training networks with billions of parameters possible.
Video transcript
- 0:01 A computational graph splits a calculation into operations:
- 0:05 multiply a by b, then add c.
- 0:08 Going backward, automatic differentiation applies the chain rule:
- 0:12 the derivative with respect to a is 2, and with respect to b, 3.
More videos
Explainer
Backpropagation: The Step-by-Step Derivation
The backpropagation derivation, factor by factor with the chain rule.
Explainer
Backpropagation: The Intuition
The intuition of backpropagation: the error flows backward through the network.
Explainer
Batch, Stochastic and Mini-Batch Gradient Descent
Batch, stochastic and mini-batch: three ways to traverse the data.
