Entropy, Cross-Entropy and KL Divergence
About this video
Entropy, cross-entropy and KL divergence, visualized.
Entropy measures the average uncertainty of a distribution in bits, cross-entropy measures the cost of coding the real data with a wrong model, and KL divergence is the gap between the two. That is why minimising cross-entropy during training is the same as minimising the KL divergence from the labels.
Video transcript
- 0:01 Cross-entropy compares the prediction with the true class, B.
- 0:04 The more probability the network gives it, the lower the loss:
- 0:08 from 1.08 to 0.13.
More videos
Explainer
Firecrawl: Web Data for Agents
If you write your own crawler: robots.txt, one queue per host and 304 Not Modified make the polite option the cheap one too.
Explainer
How to Install SearXNG with Docker
One use for your own instance: an AI agent that finds its own sources by querying SearXNG, without paying per search.
Explainer
The Computational Graph and Automatic Differentiation
The computational graph and automatic differentiation, in action.
Explainer
Backpropagation: The Step-by-Step Derivation
The backpropagation derivation, factor by factor with the chain rule.