Vector Norms (L1, L2) and Distance

A vector norm is the mathematical way to measure its length, and from that measurement come the distances a neural network uses to compare data. The two most common norms are the L1 and L2 norm: the first adds absolute values and the second applies the Pythagorean theorem. Knowing how they differ clarifies everything from how an error is computed to why regularisation keeps the weights in check. The same explanation is available in Spanish.

Key takeaways

• A norm assigns a non-negative length to a vector; it is written between double bars, like ‖x‖.
• The L1 norm adds the absolute values of the components: ‖x‖₁ = Σ|xᵢ|. It is called the Manhattan distance because it measures travel across a grid.
• The L2 norm is the square root of the sum of squares: ‖x‖₂ = √(Σ xᵢ²). It is the straight-line length, the plain Euclidean distance.
• The distance between two points is the norm of their difference, and choosing L1 or L2 changes what it means for two vectors to be close.
• In a neural network, the L1 and L2 norm shows up in loss functions and in the regularisation that prevents overfitting.

What is a vector norm?

A norm is a function that takes a vector and returns a real number representing its size. To earn the name of norm it must obey three rules: it is never negative and equals 0 only for the zero vector, it scales like the vector (double the vector and you double the norm) and it respects the triangle inequality, meaning the direct path is never longer than a detour.

With the norm comes distance. The distance between two vectors x and y is the norm of their difference, $|\mathbf{x}-\mathbf{y}|$. Changing the norm changes the map of closeness: two points that look near under one measure can be far under another. The reference text Mathematics for Machine Learning^[1] devotes a whole chapter to these geometric notions because they underpin almost any algorithm that compares examples.

L1 norm (Manhattan)

The L1 norm adds the absolute values of the vector’s components:

$$|\mathbf{x}|_1 = |x_1| + |x_2| + \dots + |x_n|$$

For the vector x = (3, 4), the L1 norm is $|3| + |4| = 7$.

It is known as the Manhattan or taxicab distance because, in a city with a grid of streets, you cannot cut through the buildings diagonally: you travel so many blocks one way and so many the other. That sum of segments is exactly the L1 norm. Its most useful trait in machine learning is that it treats every component equally and tends to push many of them all the way to zero, producing sparse solutions.

L2 norm (Euclidean)

The L2 norm is the square root of the sum of the squares of the components:

$$|\mathbf{x}|_2 = \sqrt{x_1^2 + x_2^2 + \dots + x_n^2}$$

It is the formula in this article’s hero and it is nothing but the Pythagorean theorem taken to any number of dimensions. For the same vector x = (3, 4), the L2 norm is $\sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5$.

This is the straight-line, Euclidean distance, the one a ruler would measure between two points. It is the default norm when someone talks about the length of a vector with no further qualifier, and the one libraries like NumPy^[2] use in their linalg.norm function. Unlike L1, it penalises large values much more, because every component enters squared.

Norms and regularisation

Here norms stop being geometry and start shaping training. Regularising means adding to the loss a term that penalises large weights so the network does not memorise the training set. L2 regularisation, also called weight decay, adds the squared L2 norm of the weights and shrinks them smoothly towards zero. L1 regularisation adds the L1 norm and tends to set weights exactly to zero, thereby selecting the most useful features.

As Goodfellow, Bengio and Courville put it in Deep Learning^[3], «in comparison to L2 regularization, L1 regularization results in a solution that is more sparse». That sparsity is why L1 is used when you want a model to ignore many of its inputs entirely. To see where this fits in the whole picture, it helps to review the roadmap of the mathematics of neural networks.

Numerical examples

Numbers cement intuition better than any definition. Take the vector x = (3, −4): its L1 norm is $|3| + |-4| = 7$ and its L2 norm is $\sqrt{9 + 16} = 5$. The sign changes nothing, because both norms use absolute values or squares.

With three components, for v = (1, −2, 2) the L1 norm is $1 + 2 + 2 = 5$ and the L2 norm is $\sqrt{1 + 4 + 4} = \sqrt{9} = 3$. Notice that L2 is always less than or equal to L1: here 3 versus 5. And to measure a distance, the one separating the points (1, 2) and (4, 6) is the norm of their difference (3, 4), again 7 in L1 and 5 in L2. That same play of differences and squares is what feeds the dot product and the neuron in every layer.

Frequently asked questions

What is the difference between the L1 and L2 norm?

The L1 norm adds the absolute values of the components and measures grid-like travel; the L2 norm adds the squares and takes the root, measuring the straight line. In practice, L1 favours sparse solutions with many zeros, while L2 spreads the weight more smoothly and penalises large values more heavily.

Why are norms used in neural networks?

Because they serve to measure errors and to control the weights. The distance between the prediction and the correct answer is expressed as a norm, and adding the norm of the weights to the loss stops them from growing without bound. That penalty, regularisation, is one of the most effective tools against overfitting.

Is the L2 norm the same as the Euclidean distance?

Yes when it is applied to the difference of two vectors. The L2 norm of a vector is its length from the origin; the Euclidean distance between two points is the L2 norm of the subtraction of their coordinates. They are the same formula, the square root of the sum of squares, applied to a vector or to the difference of two.

Conclusion

Norms turn a vector into a number we can compare, and choosing between L1 and L2 decides what measuring means and what being close means. L1 counts blocks and hunts for zeros; L2 draws straight lines and spreads smoothly. With those two ideas you understand both the distance between data and the regularisation of the weights. The natural next step is to review the roadmap of the mathematics of neural networks and to practise with real vectors.

Sources

1. Mathematics for Machine Learning
2. NumPy
3. Deep Learning
4. Norm (mathematics) (Wikipedia)

Route: Mathematical Foundations of Neural Networks