Entropy, Cross-Entropy and KL Divergence
About this video
Entropy, cross-entropy and KL divergence, visualized.
Entropy measures the average uncertainty of a distribution in bits, cross-entropy measures the cost of coding the real data with a wrong model, and KL divergence is the gap between the two. That is why minimising cross-entropy during training is the same as minimising the KL divergence from the labels.
Video transcript
- 0:01 Cross-entropy compares the prediction with the true class, B.
- 0:04 The more probability the network gives it, the lower the loss:
- 0:08 from 1.08 to 0.13.
More videos
Explainer
The Computational Graph and Automatic Differentiation
The computational graph and automatic differentiation, in action.
Explainer
Backpropagation: The Step-by-Step Derivation
The backpropagation derivation, factor by factor with the chain rule.
Explainer
Backpropagation: The Intuition
The intuition of backpropagation: the error flows backward through the network.
Explainer
Batch, Stochastic and Mini-Batch Gradient Descent
Batch, stochastic and mini-batch: three ways to traverse the data.