Derivatives, the Rate of Change That Teaches the Network
Table of contents
A derivative measures the rate of change of a function: how much its output responds when we nudge the input a little. That idea, which looks like a high-school technicality, is exactly what lets a neural network learn. When the network is wrong, the derivative tells it in which direction and how strongly to correct each weight. Without derivatives there would be no gradient descent and no backpropagation, and without them there would be no deep learning. The same explanation is available in Spanish.
Key takeaways
- A derivative measures the instantaneous rate of change of a function: the limit of the ratio
[f(x+h) − f(x)]/hashapproaches 0. - Geometrically, the derivative at a point is the slope of the line tangent to the curve there.
- In a neural network, the derivative of the loss with respect to each weight tells you how to change that weight to reduce the error.
- Gradient descent uses those derivatives to take small steps, scaled by the learning rate
η, in the direction that lowers the loss. - Newton and Leibniz formalised calculus in the 17th century; today the same rules tune billions of parameters.
What does a derivative measure
Picture a function f that takes a number x and returns another number. The derivative answers a very concrete question: if I increase x by a tiny amount, how much does f(x) change? The answer is a ratio between the change in output and the change in input, measured in the limit where that change becomes infinitely small.
Formally, the derivative is defined as f'(x) = lim(h→0) [f(x+h) − f(x)]/h. The numerator is how much the function rises or falls; the denominator, h, is how much we moved the input. Letting h approach 0 gives the exact rate of change at the point x, and not an average over an interval. The book Deep Learning[1] puts it this way: the derivative «specifies how to scale a small change in the input in order to obtain the corresponding change in the output».
Interpretation as slope
The most intuitive way to see a derivative is as a slope. If you draw the curve of f and trace the tangent line at a point, the derivative is the steepness of that line: how many units it rises for each unit it moves to the right. A positive derivative means the function grows; a negative one means it shrinks; a value of 0 marks a flat spot, like the top of a hill or the bottom of a valley.
The sign of the derivative tells you which way to go: positive, the function rises; negative, it falls; and a value of 0 marks a flat spot. Training looks for exactly that 0, the bottom of the valley where the error is smallest.
That last case is what training chases: we look for the bottom of the valley of the loss function, where the derivative is 0 and the error is minimal. Khan Academy[2] develops this interpretation with interactive animations that help cement the intuition.
Notation
Several notations coexist for the same thing, and it pays to recognise them all. Lagrange notation writes f'(x) (read "f prime of x"). Leibniz notation writes dy/dx, which recalls the ratio of a tiny change in y to a tiny change in x. When a function depends on several variables, we use partial derivatives, with the symbol ∂: ∂L/∂W is the derivative of the loss L with respect to a weight W, holding the rest constant.
The vector that gathers all the partial derivatives is called the gradient and is written ∇. In a network, ∇L collects the derivative of the loss with respect to each weight, and it is the compass that guides learning. Wikipedia[3] records these notations and their history in detail.
Why they matter in neural networks
A neural network computes z = Wx + b and then an activation a = f(z), layer after layer, until it produces a prediction. Comparing that prediction with the correct answer gives the loss L. Training the network means changing the weights W and biases b so that L is as small as possible.
How do we know which way to move each weight? With its derivative. The derivative ∂L/∂W says how much the loss rises if that weight rises a little. If the derivative is positive, we lower the weight; if it is negative, we raise it. Gradient descent formalises that rule: W ← W − η · ∂L/∂W, where η is the learning rate, a small number like 0.1 or 0.01 that controls the step size. Because a network chains many functions together, the chain rule shares out those derivatives layer by layer: that is backpropagation. You will find the full route in the mathematics of neural networks roadmap.
Examples
Nothing cements the idea like working it out by hand. For f(x) = x², the power rule gives f'(x) = 2x. At the point x = 3, the slope is 2 · 3 = 6: the function rises six units for each unit we advance there. At x = 0 the derivative is 0, the bottom of the parabola.
See the derivation of the derivative of x² from the limit
Start from the definition: $f'(x) = \lim_{h \to 0} \frac{(x+h)^2-x^2}{h}$. Expand the numerator: $(x+h)^2-x^2 = x^2+2xh+h^2-x^2 = 2xh+h^2$. Divide by $h$: $\frac{2xh+h^2}{h} = 2x+h$. Letting $h \to 0$ drops the last term and leaves $f'(x) = 2x$.
A second example, closer to a network: the sigmoid function σ(z) has an especially convenient derivative, σ'(z) = σ(z)·(1 − σ(z)), whose maximum value is 0.25 at z = 0. That shape explains why sigmoids saturate and slow learning when z is very large or very small. The text Neural Networks and Deep Learning[4] shows this calculation inside backpropagation step by step.
Frequently asked questions
What is the difference between a derivative and a slope?
They are almost the same: the derivative at a point is the slope of the tangent line to the curve there. The difference is that "slope" usually refers to a whole straight line, while the derivative measures the local steepness, which can change from one point to another along a curve.
What is a partial derivative?
It is the derivative of a function of several variables with respect to just one of them, treating the others as constants. In a neural network we use partial derivatives because the loss depends on thousands or millions of weights at once, and we need to know how each one contributes separately.
Why is a limit used in the definition?
Because we want the rate of change at an instant, not an average. The ratio [f(x+h) − f(x)]/h gives an average change over an interval of size h; letting h approach 0 turns that average into the exact rate at the point.
Conclusion
A derivative is nothing more than a rate of change: how much the output moves when we move the input. Seen as a slope, it tells us which way and how strongly to correct. That signal, multiplied by the learning rate and shared out with the chain rule, is what turns a neural network into a system that learns. The natural next step is to see how those derivatives combine in the mathematics of neural networks roadmap.
