The Dot Product and the Neuron
Table of contents
The dot product is the operation that turns a list of inputs and a list of weights into a single number: the neuron’s weighted sum. You multiply each input by its weight, add everything up, and get the value $z$ that the neuron will then pass to its activation function. Understanding this operation means understanding 90% of what a network does inside, because a whole layer is nothing more than many dot products at once. The same explanation is available in Spanish.
Key takeaways
- The dot product between the input vector $\mathbf{x}$ and the weight vector $\mathbf{w}$ is computed by multiplying component by component and summing: $\mathbf{w} \cdot \mathbf{x} = \sum_i w_i x_i$.
- A neuron uses that result for its weighted sum $z = \mathbf{w} \cdot \mathbf{x} + b$, where $b$ is the bias; it then applies the activation $a = f(z)$.
- Geometrically, $\mathbf{w} \cdot \mathbf{x} = \lVert\mathbf{w}\rVert\,\lVert\mathbf{x}\rVert \cos\theta$, so the dot product measures how aligned two vectors are: maximum if they point the same way, zero if they are perpendicular.
- That same formula is the basis of cosine similarity, used by search engines and recommender systems to compare vectors.
- A layer with 784 inputs and 128 neurons performs 100,352 input-weight products in a single matrix multiplication, the work that graphics cards accelerate.
What the dot product is
The dot product takes two vectors of the same size and returns a single number. The recipe is straightforward: you pair the components in the same order, multiply each pair, and add all the products. If $\mathbf{x} = [x_1, x_2, x_3]$ and $\mathbf{w} = [w_1, w_2, w_3]$, then $\mathbf{w} \cdot \mathbf{x} = w_1 x_1 + w_2 x_2 + w_3 x_3$.
That number summarises, in one figure, how similar the two vectors are, taking into account both their direction and their magnitude. It is the fundamental operation of linear algebra applied to deep learning, the same one that appears in every neuron of a network and that we already used in the sigmoid function. If you want the full map of the mathematics involved, see the guide on the mathematics behind neural networks.
How a neuron computes the weighted sum
A neuron receives several inputs and assigns each one a weight, which expresses its importance. To combine them it does nothing exotic: it computes the dot product between the input vector and the weight vector, and adds the bias $b$. The result is the pre-activation $z = \mathbf{w} \cdot \mathbf{x} + b$.
The bias shifts the threshold at which the neuron starts to respond, just like the intercept of a line. Once $z$ is computed, the neuron applies an activation function $f$ and produces its output $a = f(z)$. As Michael Nielsen wrote about the perceptron, «the neuron’s output, 0 or 1, is determined by whether the weighted sum is less than or greater than some threshold value». That threshold is exactly what the bias lets you move.
Geometric interpretation
The dot product also has a very useful geometric reading: $\mathbf{w} \cdot \mathbf{x} = \lVert\mathbf{w}\rVert\,\lVert\mathbf{x}\rVert \cos\theta$, where $\lVert\mathbf{w}\rVert$ and $\lVert\mathbf{x}\rVert$ are the lengths of the vectors and $\theta$ is the angle between them. Three clear cases follow. If the vectors point in the same direction, $\theta = 0°$, $\cos\theta = 1$ and the product is maximal. If they are perpendicular, $\theta = 90°$, $\cos\theta = 0$ and the product is zero. If they point in opposite directions, $\theta = 180°$, $\cos\theta = -1$ and the product is as negative as possible.
The sign of $\cos\theta$ alone already tells you the neuron’s stance: positive if the input aligns with the weights, negative if it opposes them, and zero if it is indifferent.
Applied to a neuron, this means the weight vector defines a preferred direction in the input space. The more the input aligns with that direction, the larger $z$ becomes and the stronger the neuron’s response. The weights are, in essence, a template against which the neuron compares each piece of data.
The dot product and similarity
Because the dot product grows when two vectors point in the same direction, it works as a measure of similarity. If you normalise the vectors so their length is 1, the dot product reduces to $\cos\theta$, and that is exactly cosine similarity. A value of 1 means vectors that are identical in direction, 0 means independence, and -1 means opposite directions.
This idea is everywhere: search engines compare the query with documents, recommender systems compare users and products, and language models compare word embeddings by measuring their dot product. The neuron makes the same comparison, but with weights that are adjusted during training rather than fixed.
A worked example
Let us see it with numbers. Suppose a neuron with three inputs $\mathbf{x} = [3, 1, 2]$, weights $\mathbf{w} = [0.4,\, -0.5,\, 0.6]$ and bias $b = 0.1$. The dot product is $\mathbf{w} \cdot \mathbf{x} = 3 \times 0.4 + 1 \times (-0.5) + 2 \times 0.6 = 1.2 – 0.5 + 1.2 = 1.9$. Adding the bias, the pre-activation is $z = 1.9 + 0.1 = 2.0$.
Notice the role of each weight. The first and third inputs add positive value because their weights are positive; the second subtracts, because its weight is negative. If that neuron uses a sigmoid activation, $f(2.0) \approx 0.88$, so the neuron fires fairly strongly. Changing a single weight moves the result: if $w_2$ went from $-0.5$ to $0.5$, the product would rise to $1.2 + 0.5 + 1.2 = 2.9$ and the pre-activation to $3.0$.
Frequently asked questions
How does the dot product differ from matrix multiplication?
The dot product operates on two vectors and returns a number. Matrix multiplication is just many dot products organised together: each element of the result is the dot product of a row by a column. That is why a neural network layer, which computes the output of many neurons at once, is written as a matrix multiplication $\mathbf{z} = W\mathbf{x} + \mathbf{b}$.
Why is it also called the scalar product?
Because the result is a scalar, that is, a single number, not another vector. The terms dot product and scalar product name the same operation. The dot in the name comes from the notation $\mathbf{w} \cdot \mathbf{x}$, where a small dot sits between the two vectors.
Conclusion
The dot product is the brick from which every neural network is built: it multiplies inputs by weights, adds them up, and produces the number the neuron will interpret. Its geometric reading explains why it measures alignment and why it is the basis of cosine similarity. The natural next step is to see what the network does with that weighted sum, applying an activation function such as the sigmoid to decide how strongly each neuron fires.
