Every artificial neuron does the same thing: it weighs its inputs, adds them up and applies an adjustment. The weights decide how much each input matters, the bias moves the result up or down, and the sum of everything is the weighted sum z = Wx + b. That number is the heart of the computation and the one the network changes as it learns. Understanding these three pieces means understanding the basic unit of deep learning. The same explanation is available in Spanish.

Key formula $z = \sum_i w_i x_i + b$

Key takeaways

  • Weights and biases are the parameters a network tunes during training; everything else is fixed.
  • A weight measures how important an input is: the larger its absolute value, the more that input influences the result.
  • The bias is an offset that lets a neuron fire even when all its inputs are zero.
  • The weighted sum z = Wx + b combines inputs, weights and bias into a single number before the activation function.
  • A model like GPT-3 holds 175 billion of these parameters, and all of them are learned with the same rule.

What weights represent

A weight is a number that multiplies an input. If a neuron receives input x and assigns it weight w, its contribution to the result is w · x. When the weight is large, that input matters a lot in the decision; when it is close to zero, it barely counts; and when it is negative, it pushes the result the other way.

With several inputs, the weights form a vector (or a matrix when there are several neurons). Computing their joint effect is a dot product between the input vector and the weight vector. Weights start out as small random values and are exactly what training gradually refines. In Deep Learning[1], Goodfellow, Bengio and Courville describe the weights as the parameters that control how the signal is transformed as it moves from one layer to the next.

The role of the bias

The bias is a number added at the end, after inputs and weights are combined. Its job is to shift the result: without it, when every input is zero the weighted sum would always be zero and the neuron would have no room to adjust its firing point.

A useful analogy is the line y = m·x + b from geometry: the weights are the slope and the bias is the intercept, the term that raises or lowers the whole line. As Michael Nielsen explains in Neural Networks and Deep Learning[2], the bias can be read as a measure of how easy it is to get the neuron to fire. Each neuron has its own bias, so a layer of 100 neurons has 100 biases, one per neuron.

The weighted sum z = Wx + b

The weighted sum brings everything together in one operation. With $n$ inputs, the neuron computes:

$$z = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b = \sum_i w_i x_i + b$$

In matrix notation, for a whole layer this is written $z = Wx + b$, where $W$ is the weight matrix, $x$ the input vector and $b$ the bias vector. That result $z$ is sometimes called the pre-activation, because it has not yet passed through the activation function $a = f(z)$. As Nielsen puts it, «a way you can think about the perceptron is that it’s a device that makes decisions by weighing up evidence». The weighted sum is exactly that act of weighing. You can see how it fits into the full route in the mathematics behind neural networks roadmap.

The bias can be absorbed into the weights with a common trick: add a constant input $x_0 = 1$ with weight $w_0 = b$. The weighted sum then becomes a single dot product $z = \sum_{i=0}^{n} w_i x_i$, with no separate term.

See the derivation

For a layer with several neurons, each neuron $j$ has its own row of weights $w_{j1}, w_{j2}, \dots, w_{jn}$ and its own bias $b_j$, and computes $z_j = \sum_i w_{ji} x_i + b_j$. Stacking the $m$ neurons, those rows form the matrix $W$ and the biases the vector $b$, so the $m$ scalar equations are written all at once as the matrix-vector product $z = Wx + b$.

How they are adjusted during training

At the start the weights and bias are random values, so the network is wrong almost every time. Training measures that error with a loss function L and computes, for every weight and bias, how much the error would change if we nudged them a little: that is the gradient ∇. It then adjusts each parameter in the direction that reduces the error, with a step proportional to the learning rate η.

The basic gradient-descent rule is w ← w - η · ∂L/∂w, and the same for the bias. This cycle (measure the error, compute gradients, adjust) repeats thousands or millions of times. The idea goes back to Frank Rosenblatt’s perceptron in 1958, which already changed weights based on its hits and misses, though with a far simpler rule than today’s.

A numerical example

Let us see it with a two-input neuron. Suppose inputs $x = [0.5,\ 0.9]$, weights $w = [0.4,\ 0.7]$ and bias $b = -0.3$. The weighted sum is:

$$z = 0.4 \cdot 0.5 + 0.7 \cdot 0.9 + (-0.3) = 0.2 + 0.63 – 0.3 = 0.53$$

If we apply the sigmoid function as the activation, $a = f(0.53) \approx 0.63$. Now imagine training raises the second weight from $0.7$ to $0.9$: the weighted sum becomes $0.71$ and the output rises to $0.67$. That small change in a single weight moves the neuron’s response, and multiplied across millions of parameters it is how a whole network learns.

Frequently asked questions

What is the difference between a weight and a bias?

A weight multiplies a specific input and measures its importance; there is one weight per input connection. The bias is a single number added at the end, independent of the inputs, and it shifts the point at which the neuron fires.

Why does a neuron need a bias?

Without a bias, if all inputs are zero the weighted sum would be zero and the neuron would be stuck at that point. The bias gives it freedom to fire earlier or later, just as the intercept shifts a line up or down.

Are weights and biases set by hand?

No. They are initialised with small random values and then the training algorithm adjusts them on its own, guided by the gradient of the loss function. The programmer defines the architecture and the data, not the final value of each parameter.

Conclusion

Weights, biases and the weighted sum are the most basic mechanics of deep learning: weights weigh the inputs, the bias shifts the result, and z = Wx + b gathers it all into one number. Adjusting those parameters millions of times is, quite literally, what it means for a network to learn. The natural next step is to see what the activation function does with that z and how the gradient shares the error across all the weights.

Sources

  1. Deep Learning
  2. Neural Networks and Deep Learning
  3. Artificial neuron (Wikipedia)
  4. Perceptron (Wikipedia)

Route: The Neuron and Activation Functions