Two inverse functions, the exponential eˣ and the natural logarithm ln(x), underpin much of the mathematics of deep learning. The exponential shows up whenever a network turns numbers into probabilities, inside the sigmoid and the softmax. The natural logarithm shows up on the opposite side, when we measure the error with cross-entropy. Understanding these two pieces well, and how they cancel each other out, clarifies why the loss formulas take the shape they do. The same explanation is available in Spanish.

Key formula $e^x \quad \text{and} \quad \ln(x)$

Key takeaways

  • The exponential and logarithm are inverse functions: ln(eˣ) = x and e^(ln x) = x, and that relationship is what simplifies the derivatives during training.
  • The number e ≈ 2.718 is the natural base because the derivative of eˣ is eˣ itself, something no other base achieves.
  • The exponential builds the sigmoid and softmax, which turn arbitrary scores into probabilities between 0 and 1.
  • The natural logarithm defines cross-entropy, the standard loss function for classification.
  • The log-sum-exp trick prevents numerical overflow when computing exponentials of large numbers, a real problem in any network.

The number e and the exponential function

The exponential function $e^x$ is built on the number $e \approx 2.718281828$, an irrational constant every bit as fundamental as $\pi$. Its most useful property for deep learning is surprising: the derivative of $e^x$ is $e^x$ again. It is the only function (up to a constant factor) that is its own derivative, and that is why calculus with exponentials comes out so clean.

Two more properties matter. The exponential is never negative: $e^x > 0$ for any $x$, which makes it ideal for producing probabilities. And it turns sums into products, $e^{a+b} = e^a \cdot e^b$, an identity that reappears when differentiating the softmax. With concrete values: $e^0 = 1$, $e^1 \approx 2.718$ and $e^{-1} \approx 0.368$. As $x$ grows toward infinity, $e^x$ explodes, and as $x$ tends to minus infinity, $e^x$ approaches 0 without ever reaching it. That asymmetry is the basis of the S-shaped curve we study in the sigmoid function.

The natural logarithm

The natural logarithm $\ln(x)$ is the inverse of the exponential: it undoes what $e^x$ does. If $y = e^x$, then $x = \ln(y)$. That is why $\ln(e) = 1$, $\ln(1) = 0$ and $\ln$ only accepts positive numbers, exactly the range the exponential produces.

Its derivative is also remarkably simple: the derivative of $\ln(x)$ is $\frac{1}{x}$. This cleanliness is why the natural logarithm, not the base-10 logarithm, is the one that appears in loss functions. The logarithm turns products into sums, $\ln(a \cdot b) = \ln(a) + \ln(b)$, which lets us convert the multiplication of many small probabilities (which drifts toward 0 and causes numerical errors) into a stable sum of logarithms. That switch from product to sum is why we almost always work with the log-likelihood rather than the raw likelihood.

Why they appear in sigmoid, softmax and cross-entropy

The sigmoid is defined as

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

The exponential in the denominator is what squashes any real number into the interval $(0, 1)$ so we can read it as a probability. Its multi-class sibling is the softmax,

$$\mathrm{softmax}(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$$

which exponentiates each score and then normalises so the sum is 1. Here the positivity of the exponential is key: it guarantees no probability comes out negative.

The logarithm enters when measuring the error. Cross-entropy for classification is written $L = -\sum_i y_i \ln(a_i)$, where $a_i$ is the probability the network predicts and $y_i$ the true label. If the network assigns probability 1 to the correct class, $\ln(1) = 0$ and the loss is zero; if it assigns a tiny probability, the logarithm blows up and punishes the mistake hard. The magic happens when softmax meets cross-entropy: the logarithm of the numerator cancels the exponential of the softmax, and the gradient collapses to the clean difference $a_i-y_i$. That result, which ties back to the mathematics of neural networks roadmap, is what makes training classifiers so efficient.

When softmax and cross-entropy act together, the exponential and the logarithm cancel and the gradient collapses to $a_i-y_i$: the difference between predicted and true. That cancellation is what makes training numerically stable and cheap to compute.

Numerical stability (log-sum-exp)

There is a practical problem: $e^z$ with large $z$ overflows memory. In 64-bit floating point, $e^{710}$ already returns infinity, and a network can easily produce scores that size. The solution is the log-sum-exp trick. As the reference book by Goodfellow, Bengio and Courville puts it, «the softmax can saturate when the differences between the inputs become extreme».

The idea is to subtract the maximum before exponentiating. Instead of computing $\sum_j e^{z_j}$ directly, you compute $m = \max(z)$ and then $m + \ln!\left(\sum_j e^{z_j-m}\right)$. The mathematical result is identical, but now the largest exponent is $e^0 = 1$, so there is never overflow. Every serious library (PyTorch, TensorFlow) implements softmax and cross-entropy in this combined, stable form rather than chaining the two operations separately.

See why the trick does not change the result

Start from $\ln\!\left(\sum_j e^{z_j}\right)$ and factor $e^m$ out of the sum: $\sum_j e^{z_j} = \sum_j e^{m}\,e^{z_j-m} = e^{m}\sum_j e^{z_j-m}$. Taking the logarithm and using $\ln(a \cdot b) = \ln(a) + \ln(b)$: $\ln\!\left(\sum_j e^{z_j}\right) = m + \ln\!\left(\sum_j e^{z_j-m}\right)$. The value is identical, but the largest exponent becomes $e^0 = 1$.

Examples

A numerical case helps show the mechanism. Suppose three output scores $z = [2, 1, 0]$. We exponentiate: $e^2 \approx 7.389$, $e^1 \approx 2.718$ and $e^0 = 1$. The sum is $11.107$, and the softmax probabilities come out roughly $0.665$, $0.245$ and $0.090$, which add up to 1. The highest-scoring class takes the largest probability, as expected.

Now the loss. If the correct label was the first class, cross-entropy is $-\ln(0.665) \approx 0.408$. Had the network been more confident, say 0.95, the loss would drop to $-\ln(0.95) \approx 0.051$. And had it been completely wrong, assigning 0.01 to the correct class, the loss would rise to $-\ln(0.01) \approx 4.605$. That contrast between 0.05 and 4.6 shows how the natural logarithm amplifies the penalty for serious errors without any artificial rules.

Frequently asked questions

Why use the natural logarithm and not the base-10 logarithm?

Because its derivative is 1/x, the simplest possible, and because it cancels exactly with the exponential in the sigmoid and softmax. Using base 10 would only introduce a constant factor of ln(10) ≈ 2.303 into every calculation, with no benefit at all.

What is the exact relationship between the exponential and the logarithm?

They are inverse functions: each undoes the other. It holds that ln(eˣ) = x for every x, and e^(ln y) = y for every positive y. That cancellation is what simplifies the gradients when softmax and cross-entropy act together.

Is exponential overflow a real danger in practice?

Yes, it is a real problem. Without the log-sum-exp trick, an input score greater than about 710 already produces infinity in 64-bit floating point and breaks training. That is why libraries implement softmax in a numerically stable way out of the box.

Conclusion

The exponential and the natural logarithm are not mathematical decorations but two sides of one coin that runs through all of deep learning. The exponential builds the probabilities with the sigmoid and softmax, the logarithm evaluates them with cross-entropy, and their mutual cancellation is what makes training elegant and efficient. The natural next step is to review how they fit into the softmax function inside a full classifier.

Sources

  1. Deep Learning (Goodfellow, Bengio and Courville)
  2. Neural Networks and Deep Learning (Michael Nielsen)
  3. Exponential function (Wikipedia)
  4. Cross-entropy (Wikipedia)

Route: Mathematical Foundations of Neural Networks