Every neural network learns by differentiating: it needs to know how the error changes when it nudges each weight. Those derivatives do not appear by magic; they come from applying a handful of differentiation rules taught in a first calculus course. Master those 5 or 6 rules and you can follow, step by step, how a network computes its gradient and adjusts its weights with gradient descent. This guide reviews them one by one with examples. The same explanation is available in Spanish.

Key formula $(fg)’ = f’g + fg’$

Key takeaways

  • The differentiation rules are fixed formulas that turn a function into its derivative without going back to the limit every time.
  • The power rule and the product and quotient rules cover most of the algebraic expressions that show up inside a neuron.
  • The derivatives of the exponential and the logarithm underpin heavily used functions such as the sigmoid, the softmax and cross-entropy.
  • These rules combine with the chain rule to propagate the error layer by layer: that is backpropagation.
  • With just 6 basic rules you can differentiate by hand almost any activation or loss function in a small network.

The power rule

The first and most used. If a function is a power, $f(x) = x^n$, its derivative is $f'(x) = nx^{n-1}$: bring the exponent down as a coefficient and subtract 1 from the exponent.

$$\frac{d}{dx}\,x^n = nx^{n-1}$$

So the derivative of $x^2$ is $2x$, that of $x^3$ is $3x^2$ and that of $x^5$ is $5x^4$.

The rule works with any real exponent, not just integers. The square root is $x$ raised to $0.5$, and its derivative is $0.5\,x^{-0.5}$. A multiplying constant is preserved, and the derivative of a sum is the sum of the derivatives, so a polynomial is differentiated term by term. With those 3 ideas together you can already differentiate any polynomial that appears inside a network. You can review the full reasoning at Khan Academy[1].

The product and quotient rules

When two functions are multiplied, multiplying their derivatives is not enough. The product rule says $(fg)’ = f’g + fg’$: the derivative of the first times the second undifferentiated, plus the first undifferentiated times the derivative of the second. It is the formula on the cover of this article and one of the most often forgotten.

For a quotient, the quotient rule extends the idea:

$$\left(\frac{f}{g}\right)’ = \frac{f’g-fg’}{g^2}$$

It appears, for instance, when you differentiate the sigmoid function, which is a quotient with an exponential in the denominator. If you prefer a formal proof, Wikipedia[2] collects both the product and the quotient rules with their proofs.

In the quotient rule the order matters: the numerator is $f’g-fg’$, not the other way round. Swapping those two terms flips the sign of the result and is one of the most common mistakes when differentiating the sigmoid.

Derivatives of common functions

Besides powers, it pays to memorise a few reference derivatives because they come up again and again:

  • The derivative of a constant is 0: a constant does not change.
  • The derivative of $x$ is 1.
  • The derivative of sine is cosine, and that of cosine is minus sine.
  • The derivative of ReLU, $\max(0, x)$, is 1 when $x$ is positive and 0 when it is negative.

That last one is why ReLU is so convenient in deep learning: its derivative is trivial to compute, just a 1 or a 0, which speeds up every training step compared with costlier activations.

The exponential and the logarithm

Here are the two derivatives that matter most in a neural network. The exponential function $e^x$ is special because it is its own derivative: $(e^x)’ = e^x$. That property is what gives the sigmoid and the softmax such clean, cheap gradients.

The natural logarithm is its companion: $(\ln x)’ = \dfrac{1}{x}$. It shows up whenever a network uses cross-entropy as its loss function, because that loss is built from logarithms of probabilities. As the Deep Learning[3] textbook by Goodfellow, Bengio and Courville puts it, «the chain rule of calculus is used to compute the derivatives of functions formed by composing other functions whose derivatives are known». Exponential, logarithm and chain rule together explain almost all the machinery of training.

Worked examples

Let us see how the pieces fit inside a neuron. A layer computes $z = Wx+b$ and then applies an activation $a = f(z)$. To train it we need the derivative of $a$ with respect to $z$, and that is where these rules come in.

If the activation is the sigmoid, $f(z) = \dfrac{1}{1 + e^{-z}}$, you combine the quotient rule and the exponential rule. The result is surprisingly simple: $f'(z) = f(z)\,(1-f(z))$. The derivative is expressed through the output itself, so the network reuses it without recomputing anything.

See the derivation of the sigmoid’s derivative

Write the sigmoid as $f(z) = (1+e^{-z})^{-1}$ and apply the chain rule together with $(e^{-z})’ = -e^{-z}$: $f'(z) = -(1+e^{-z})^{-2}\cdot(-e^{-z}) = \dfrac{e^{-z}}{(1+e^{-z})^2}$. Now rewrite $e^{-z} = (1+e^{-z})-1$ to split the fraction: $f'(z) = \dfrac{1}{1+e^{-z}}\cdot\dfrac{(1+e^{-z})-1}{1+e^{-z}} = f(z)\,(1-f(z))$.

The same happens with the loss. Differentiating cross-entropy applies the derivative of the logarithm, $\dfrac{1}{x}$, and to propagate the error backwards you multiply all these derivatives following the chain rule. That chained product is the gradient the optimiser uses, with a learning rate $\eta$, to update each weight $W$. To see where each block fits, this article is part of the roadmap of the mathematics behind neural networks.

Frequently asked questions

How many differentiation rules do I need to understand a neural network?

Six are enough: the power rule, the product rule, the quotient rule, the exponential rule, the logarithm rule and the chain rule. With them you can differentiate by hand almost any activation or loss function in a small network.

Why is the exponential so important in deep learning?

Because e^x is its own derivative, so functions built from it (the sigmoid and the softmax) have gradients that are very easy to compute. That lowers the cost of each training step, which is decisive when a network repeats the calculation millions of times.

How do these rules relate to backpropagation?

Backpropagation is nothing more than applying the chain rule from back to front, multiplying the local derivatives of each layer. Each of those local derivatives is obtained with the basic rules this guide reviews.

Conclusion

The differentiation rules are not an obstacle but a short set of formulas that unlocks how a network learns. The power rule, the product and quotient rules, and the exponential and logarithm rules cover almost everything you need; the chain rule ties them together. The natural next step is to see how they chain up in backpropagation within the full mathematics roadmap.

Sources

  1. Khan Academy
  2. Wikipedia
  3. Deep Learning

Route: Mathematical Foundations of Neural Networks