Probability is the language a neural network uses to express its uncertainty. When a network classifies an image, it does not answer with a firm yes but with a distribution: 80% cat, 15% dog, 5% other. Understanding that distribution, the value we expect from it and how to measure whether the model is right lets you see where the softmax function and cross-entropy come from. This guide walks through those ideas from scratch. The same explanation is available in Spanish.

Key formula $\mathbb{E}[X] = \sum_i x_i\,P(x_i)$

Key takeaways

  • Probability in neural networks describes uncertainty: each output is a distribution over possible outcomes, not a single number.
  • A probability distribution assigns each outcome a weight between 0 and 1, and all those weights sum to exactly 1.
  • The expected value $\mathbb{E}[X] = \sum_i x_i\,P(x_i)$ averages the outcomes, weighting each one by its probability.
  • The softmax function turns the network’s scores into a valid categorical distribution, ready to be read as probabilities.
  • Maximum likelihood is the principle that justifies cross-entropy: training means making the observed data more probable.

Probability, distributions and expected value

A probability is a number between 0 and 1 that measures how believable an event is: 0 is impossible and 1 is certain. When we roll a fair die, each face has probability 1/6, roughly 0.167. The set of all those probabilities forms a distribution, and its golden rule is that the sum over all cases equals 1.

The expected value condenses a distribution into a single number. It is computed as $\mathbb{E}[X] = \sum_i x_i\,P(x_i)$, the sum of each outcome multiplied by its probability. For the die, the expected value is (1+2+3+4+5+6)/6 = 3.5, even though 3.5 never shows up on a real roll. In a neural network, this same idea appears when we average the error over many examples: the loss function is, at heart, an expected value. The book Deep Learning[1] devotes its chapter 3 to exactly these foundations before building any model.

When a network predicts, we almost always work with conditional probability: $P(y \mid x)$, the probability of label $y$ given the input $x$. That symbol is the heart of supervised classification.

Categorical distributions and softmax

When the problem has several classes (cat, dog, bird), the output is a categorical distribution: a vector of probabilities, one per class, summing to 1. The catch is that the network’s last layer produces unconstrained numbers, called logits, which can be negative or greater than 1.

The softmax function fixes that mismatch. It takes the logit vector $z = Wx + b$ and transforms it by exponentiating each component and dividing by the total sum, so the result is always positive and sums to 1. If the logits are 2.0, 1.0 and 0.1, softmax returns roughly 0.66, 0.24 and 0.10: a legitimate distribution. We explain it in detail in the softmax function, a key piece for multi-class classification.

Likelihood and maximum likelihood

Likelihood flips the usual question around. Instead of asking how probable a data point is, it asks: given a set of observed data, how well does this model with these weights explain them? The likelihood is the probability the model assigns to the real data.

The principle of maximum likelihood says the best weights are those that make the observed data most probable. Because multiplying many small probabilities gives tiny numbers, we work with the logarithm: maximising the log-likelihood is equivalent and numerically stable. By convention, in optimisation we prefer to minimise, so we minimise the negative log-likelihood. As the book Mathematics for Machine Learning[2] puts it: «maximum likelihood estimation seeks the model parameters that maximise the probability of the observed data».

Multiplying many small probabilities yields tiny, numerically unstable numbers. That is why we work with the logarithm: it turns the product $\prod_i P(y_i \mid x_i)$ into the sum $\sum_i \log P(y_i \mid x_i)$, far easier to handle and with the same maximum.

Relation to loss functions

Here the circle closes. Minimising the negative log-likelihood of a categorical distribution is exactly computing the cross-entropy between the network’s prediction and the correct label. In other words, the cross-entropy loss we use daily for classification is not an arbitrary choice: it follows directly from the maximum likelihood principle.

See the derivation: from likelihood to cross-entropy

The likelihood of a set of independent data points is the product of the probabilities the model assigns to each correct label: $\prod_i P(y_i \mid x_i)$. Taking the negative logarithm turns the product into a sum: $-\log \prod_i P(y_i \mid x_i) = -\sum_i \log P(y_i \mid x_i)$. That last term is exactly the cross-entropy between the predicted distribution and the real label, which is why minimising the negative log-likelihood is equivalent to minimising the cross-entropy.

The full flow fits together like this: the network produces logits, softmax turns them into probabilities $P(y \mid x)$, cross-entropy measures how far that distribution is from the correct answer, and gradient descent adjusts the weights $W$ to reduce the loss. Probability is not an add-on: it is what gives meaning to the entire objective function. Cross-entropy[3] measures, in bits or nats, the distance between two distributions.

Examples

A numerical example helps fix the ideas. Suppose a network classifies among 3 classes and, for a cat image (the correct class), predicts the distribution 0.7, 0.2 and 0.1. The cross-entropy loss is simply the negative logarithm of the probability assigned to the correct class: -ln(0.7) ≈ 0.357. Had the network been more confident, with 0.95 on the correct class, the loss would drop to -ln(0.95) ≈ 0.051, almost seven times smaller. And had it been wrong with 0.1, the loss would rise to -ln(0.1) ≈ 2.303. Thus the loss strongly punishes misplaced confidence.

A second example, the expected value of a bet: if you win 10 euros with probability 0.3 and lose 3 euros with probability 0.7, the expected value is

$$\mathbb{E}[X] = 10 \times 0.3 + (-3) \times 0.7 = 0.9 \text{ euros per play.}$$

The same computation, $\mathbb{E}[X] = \sum_i x_i\,P(x_i)$, is what averages the loss over a training batch.

Frequently asked questions

What is the difference between probability and likelihood?

Probability fixes the model and asks how probable the data are; likelihood fixes the data and asks how good the model parameters are. Numerically they use the same formula, but what we hold fixed and what we vary changes. Training a network is a likelihood exercise: we search for the weights that best explain the examples.

Why is softmax used before cross-entropy?

Because softmax guarantees the outputs are a valid distribution (positive and summing to 1), and cross-entropy only makes sense between distributions. Together they form the standard classification block: softmax to produce $P(y \mid x)$ and cross-entropy to measure the error against the real label.

Do I need advanced statistics to train networks?

No. Understanding what a distribution is, the expected value and the idea of likelihood covers 90% of what you need. Advanced concepts (continuous distributions, Bayesian inference) enrich the intuition, but they are not essential to start classifying with neural networks.

Conclusion

Probability gives neural networks their vocabulary of uncertainty: distributions to describe what is possible, expected value to average it and likelihood to measure the fit. From those three notions spring the softmax function and cross-entropy, two tools you will use in almost any classifier. The natural next step is to see how the mathematics behind neural networks roadmap connects this piece with linear algebra and calculus.

Sources

  1. Deep Learning
  2. Mathematics for Machine Learning
  3. Cross-entropy
  4. Neural Networks and Deep Learning (Michael Nielsen)

Route: Mathematical Foundations of Neural Networks