What Is a Loss Function and a Cost Function
Table of contents
- Key takeaways
- What a loss function measures
- Loss versus cost
- Regression versus classification
- How it guides training
- The most common losses
- Frequently asked questions
- What is the difference between a loss function and a cost function?
- Which loss function should I use?
- Why must the loss be differentiable?
- Conclusion
- Sources
A loss function puts a number on the error of a prediction: how far what the network says is from what it should say. The cost function takes that number and averages it across every training example. With that single figure, the network knows whether it is doing well or badly and in which direction to correct its weights. Understanding the difference between loss and cost, and knowing which to pick, is the foundation of all training. The same explanation is available in Spanish.
Key takeaways
- A loss function measures the error on a single example; the cost function averages that loss over the whole dataset.
- The goal of training is a single number: minimise the cost by adjusting the weights with gradient descent.
- For regression, the usual choice is mean squared error; for classification, cross-entropy.
- The loss must be differentiable, because learning needs its gradient to know which way to move.
- Choosing the wrong loss makes the network optimise the wrong objective, however good the architecture is.
What a loss function measures
Imagine a network predicts a flat’s price at 200,000 euros when the real price was 250,000. The error is 50,000 euros. A loss function turns that distance into a number training can use. In compact notation we write $L(\hat{y}, y)$, where $\hat{y}$ is the network’s prediction and $y$ the correct value.
That number has two important properties. First, it equals 0 for a perfect hit and grows as the prediction drifts away. Second, it must be differentiable: training computes its gradient to know how to change each weight. A loss you cannot differentiate leaves the network without a compass. As the Wikipedia entry on loss functions[1] notes, all of decision statistics is built on this idea of penalising errors.
The exact shape of the penalty matters. Squaring the error punishes large mistakes far more than small ones, whereas using the absolute value treats them proportionally. That decision changes what the network learns to prioritise.
Loss versus cost
The two terms are used almost interchangeably, but it helps to separate them. The loss is the error on a single example: one prediction, one answer. The cost is the average loss over a batch or over the entire training set. If you have 1,000 examples, you compute 1,000 losses and average them into one cost figure:
$$J = \frac{1}{n}\sum_{i=1}^{n} L(\hat{y}_i, y_i)$$
Many authors call that average the objective function, because it is literally the objective the optimisation tries to minimise. The distinction is not pedantic: gradient descent works on the cost, not on a lone loss, which is why batch size affects training stability. A batch of 32 examples gives a noisier cost than one of 512, but it is also cheaper to compute.
Regression versus classification
The type of problem decides the loss family. In regression we predict a continuous number (a price, a temperature), and the reference is mean squared error. Its formula averages the square of the differences between prediction and true value:
$$\mathrm{MSE} = \frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i – y_i)^2$$
Squaring the error makes a miss of 10 weigh four times as much as one of 5, not twice. That is why MSE is so sensitive to outliers, and when they are common the MAE is often the better choice.
In classification we predict a category (spam or not, dog or cat), and the output is read as a probability between 0 and 1. Here the reference loss is cross-entropy, which harshly penalises confident mistakes: if the network assigns a probability of $0.01$ to the correct class, the loss spikes. The PyTorch documentation on loss functions[2] offers more than twenty variants, but almost all real work is done with these two families.
How it guides training
The loss is not a passive diagnosis but the engine of learning. The cycle is always the same: the network makes a forward prediction, the cost function measures the error, and backpropagation computes the gradient of that cost with respect to each weight. Then gradient descent takes a small step in the direction that reduces the cost.
That step is controlled by the learning rate, a typical value of $0.01$ that decides how far the weights move at each iteration. Repeated thousands of times over batches of data, the cost drops and the network improves. As Goodfellow, Bengio and Courville summarise in their reference book, «any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution defined by the data and the distribution defined by the model». That link with probability is what gives cross-entropy its theoretical footing.
The most common losses
These are the ones that show up again and again in practice:
- Mean squared error (MSE): standard regression; squares the differences.
- Mean absolute error (MAE): regression that is more robust to outliers.
- Binary cross-entropy: two-class classification with a sigmoid output.
- Categorical cross-entropy: multi-class classification with a softmax output.
- Huber: a hybrid that combines MSE near the target and MAE on large errors.
The choice is not a minor detail. Tools like scikit-learn[3] document dozens of evaluation metrics, but the loss minimised during training must match the nature of the problem and the activation function of the final layer.
Frequently asked questions
What is the difference between a loss function and a cost function?
The loss function measures the error on a single example, whereas the cost function is the average of that loss over a batch or the whole dataset. Training minimises the cost, not an individual loss.
Which loss function should I use?
It depends on the problem. For regression (predicting a number) the starting point is mean squared error. For classification (predicting a category) the standard choice is cross-entropy, paired with a sigmoid or softmax output.
Why must the loss be differentiable?
Because learning relies on gradient descent, which needs the derivative of the loss with respect to each weight to know which way to correct. Without a gradient, the network would have no systematic way to improve.
Conclusion
The loss function and the cost function are the measuring stick of deep learning: the first judges a prediction, the second summarises performance over the data and sets the number to reduce. Choosing the right loss (MSE for regression, cross-entropy for classification) aligns the network with the real problem. The natural next step is to see how that number is minimised in the mathematics of neural networks roadmap.
