Entropy, Cross-Entropy and KL Divergence

Entropy measures the uncertainty of a distribution, and KL divergence measures how much you lose by describing that distribution with a different one. These are three linked ideas from information theory: entropy, cross-entropy and KL divergence. They show up together every time a neural network computes its loss, because training a classifier is, quite literally, pushing the distribution the model predicts towards the real distribution of the labels. Understanding how the three relate makes it clear why cross-entropy is the default cost function across almost all classification. The same explanation is available in Spanish.

Key takeaways

• Entropy $H(P) = -\sum P\log P$ measures the average uncertainty of a distribution; a fair coin has 1 bit and a 6-sided die about $2.585$ bits.
• Cross-entropy $H(P,Q) = -\sum P\log Q$ measures the cost of coding data that follows $P$ using a wrong model $Q$.
• KL divergence $D_{\mathrm{KL}}(P|Q) = \sum P\log\frac{P}{Q}$ is exactly the gap: the extra bits you spend for using $Q$ instead of $P$.
• The identity that ties it together is $H(P,Q) = H(P) + D_{\mathrm{KL}}(P|Q)$: cross-entropy equals entropy plus divergence.
• Because $H(P)$ does not depend on the model, minimising cross-entropy during training is the same as minimising the KL divergence from the labels.

What is entropy (Shannon)?

Entropy was born in 1948 with Claude Shannon’s paper «A Mathematical Theory of Communication»^[1], the text that founded information theory. As Shannon wrote, «the fundamental problem of communication is that of reproducing at one point either exactly or approximately a message selected at another point». To measure how much information must be transmitted, he defined the entropy of a probability distribution $P$:

$$H(P) = -\sum P\log P$$

When the logarithm is base 2, entropy is measured in bits: the average number of bits needed to code each outcome if we use the best possible code. A fair coin, with two equally likely outcomes, has an entropy of exactly 1 bit. A balanced 6-sided die rises to $\log_2(6) \approx 2.585$ bits, because there is more uncertainty to resolve. A biased coin that lands heads 90% of the time drops to about $0.469$ bits: it is more predictable, so it informs less. Entropy is maximal when everything is equally likely and minimal (zero) when the outcome is certain. You can review the logarithm in exponential and natural logarithm in deep learning, and the wider framework in the mathematics of neural networks roadmap.

Cross-entropy explained

Entropy assumes we know the true distribution $P$. But a model almost never knows it: it works with its own estimate $Q$. Cross-entropy measures what happens when we code data that actually follows $P$ using the optimal code for $Q$:

$$H(P,Q) = -\sum P\log Q$$

If $Q$ is right and matches $P$, cross-entropy equals entropy and nothing is wasted. If $Q$ is wrong, cross-entropy is larger: we spend more bits than needed. This is exactly the number a network minimises when classifying. The network produces a distribution $Q$ (for example, with a softmax function) and compares it with the real label distribution. The two-class case is binary cross-entropy, which comes from the sigmoid. The visual intuition for why the logarithm counts bits is told very well in Visual Information Theory by Christopher Olah^[2].

KL divergence

If cross-entropy is always greater than or equal to entropy, the natural question is by how much. That gap is the Kullback-Leibler divergence, proposed in 1951:

$$D_{\mathrm{KL}}(P|Q) = \sum P\log\frac{P}{Q}$$

It measures the extra bits you spend for using model $Q$ when reality is $P$. It has three properties worth memorising. First, it is never negative: $D{\mathrm{KL}}(P|Q) \ge 0$ always, a result known as Gibbs’ inequality. Second, it equals zero only when $Q$ is identical to $P$, so it acts as a measure of how far the model is from the truth. Third, it is not symmetric: $D{\mathrm{KL}}(P|Q)$ almost never equals $D_{\mathrm{KL}}(Q|P)$, which is why it is not a distance in the strict sense.

An example clarifies the asymmetry. Let $P = [0.5,\ 0.5]$ and $Q = [0.9,\ 0.1]$. In bits, $D{\mathrm{KL}}(P|Q) \approx 0.737$, whereas $D{\mathrm{KL}}(Q|P) \approx 0.531$. They are different values, so the order of the arguments matters. The KL divergence article on Wikipedia^[3] collects these properties and their uses in statistics.

How the three relate

Here everything fits. If we expand the KL divergence by splitting the logarithm of the ratio, the central identity appears:

$$D_{\mathrm{KL}}(P|Q) = \sum P\log\frac{P}{Q} = -H(P)+H(P,Q)$$

Rearranging: $H(P,Q) = H(P) + D_{\mathrm{KL}}(P|Q)$. In words, cross-entropy is entropy plus KL divergence. Entropy is the unavoidable cost of the real uncertainty; KL divergence is the extra cost of having an imperfect model. Cross-entropy is the sum of both.

The reference book Deep Learning by Goodfellow, Bengio and Courville^[4] presents these three quantities in exactly this order in its information-theory chapter, because it is the cleanest way to derive the loss functions.

Use in deep learning

Now the practical consequence. When training a classifier, $P$ is the real label distribution and $Q$ is the one the network predicts. Because the label usually comes in one-hot form (a 1 in the correct class and zeros elsewhere), its entropy $H(P)$ is exactly 0: there is no uncertainty in the training example. Since $H(P)$ is a constant that does not depend on the weights, minimising $H(P,Q)$ is identical to minimising $D_{\mathrm{KL}}(P|Q)$. So even when libraries talk about "cross-entropy loss", under the covers you are pushing the KL divergence towards zero.

This equivalence also explains why cross-entropy punishes confident mistakes hard: if the model assigns a tiny probability to the correct class, the term $-\log Q$ shoots up. The resulting gradient is clean and stable, which makes it the default choice over alternatives such as mean squared error. The same machinery scales to huge models: a language model is trained by minimising cross-entropy over billions of tokens, one token at a time.

Frequently asked questions

why is KL divergence not a distance?

Because it is not symmetric: $D{\mathrm{KL}}(P|Q)$ and $D{\mathrm{KL}}(Q|P)$ give different values, as in the example ($0.737$ versus $0.531$ bits). A mathematical distance requires symmetry and the triangle inequality, and KL divergence satisfies neither. It is a directed measure of dissimilarity, not a metric.

what is the difference between cross-entropy and KL divergence?

Cross-entropy $H(P,Q)$ includes the background entropy $H(P)$; KL divergence measures only the excess, $H(P,Q)-H(P)$. When the labels are one-hot, $H(P) = 0$ and the two coincide numerically, which is why in network training they are used almost as synonyms even though conceptually they are different things.

what units are these quantities measured in?

It depends on the base of the logarithm. With base-2 logarithms they are measured in bits, which is standard in information theory. With natural logarithms they are measured in nats, which is what almost every deep learning library uses internally for computational convenience. The shape of the formula does not change, only the scale.

Conclusion

Entropy, cross-entropy and KL divergence are three pieces of the same story: entropy measures the real uncertainty, cross-entropy measures the cost of an imperfect model, and KL divergence is the gap between them. The identity $H(P,Q) = H(P) + D_{\mathrm{KL}}(P|Q)$ ties them together and, because $H(P)$ is constant during training, minimising one is minimising the other. The natural next step is to see how this idea takes shape in a classifier’s output with the softmax function.

Sources

1. «A Mathematical Theory of Communication»
2. Visual Information Theory by Christopher Olah
3. KL divergence article on Wikipedia
4. Deep Learning by Goodfellow, Bengio and Courville

Route: Loss Functions in Neural Networks