Scalars, Vectors, Matrices and Tensors Explained
Table of contents
- Key takeaways
- What is a scalar, a vector, a matrix and a tensor?
- How data and weights are represented
- Dimensions and shape
- Basic operations
- Example in a layer of neurons
- Frequently asked questions
- Is a tensor the same as a multidimensional array?
- Why are vectors and matrices used in neural networks?
- Do I need to master linear algebra before starting?
- Conclusion
- Sources
Scalars, vectors, matrices and tensors are the four shapes a neural network uses to store numbers. A scalar is a single number, a vector a row of numbers, a matrix a table and a tensor the same idea taken to three or more dimensions. Learning to tell them apart is the first step to understanding the vectors and matrices in neural networks, because every piece of data and every weight lives inside one of these structures. The same explanation is available in Spanish.
Key takeaways
- A scalar has 0 dimensions, a vector 1, a matrix 2 and a tensor any number: they are the staircase of linear algebra.
- Input data is represented as vectors and the weights of each layer as matrices, which is why we talk about vectors and matrices in neural networks.
- The shape of a tensor is the list of its sizes per dimension, and matching those shapes is half the work of programming a network.
- The central operation of a layer is
z = Wx + b, a matrix-by-vector multiplication plus a bias. - Graphics cards exist to multiply matrices fast, and that is why they are the engine of deep learning.
What is a scalar, a vector, a matrix and a tensor?
The simplest way to order them is by counting their dimensions, what linear algebra calls the rank of the object.
- A scalar is a single number, like the temperature
21or one specific weight. It has 0 dimensions. - A vector is an ordered list of numbers, like $\mathbf{v} = [1.7,\ 0.3,\ 2.0]$. It has 1 dimension and represents a point or a direction in a space.
- A matrix is a rectangular table of numbers with rows and columns. It has 2 dimensions and usually holds the weights connecting two layers.
- A tensor generalises the idea to 3 or more dimensions. A colour image is a 3-axis tensor (height, width and colour channels), and a batch of 32 images is a 4-axis tensor.
The name of deep learning’s most famous library, TensorFlow, comes precisely from the fact that data flows as tensors through a graph of operations. The term tensor comes from physics and geometry; you can read its formal definition on Wikipedia[1].
How data and weights are represented
Everything that enters a network becomes numbers. A sentence turns into a vector of identifiers, and a 28-by-28-pixel black-and-white image is flattened into a vector of 784 values, one per pixel. That vector is the input x.
The weights connecting a layer of $n$ inputs to another of $m$ neurons form a matrix $\mathbf{W}$ with $m$ rows and $n$ columns. In compact notation, the input satisfies $\mathbf{x} \in \mathbb{R}^{n}$ and the weights $\mathbf{W} \in \mathbb{R}^{m \times n}$. Multiplying that matrix by the vector and adding the bias $\mathbf{b}$ produces the layer’s output, the same computation we study in the mathematics behind neural networks. The dot product between each row of W and the vector x is the operation that drives each neuron.
Dimensions and shape
A tensor’s shape is the list of its sizes in each dimension. In the reference numerical library, a vector of 784 values has shape (784,), a weight matrix of 128 by 784 has shape (128, 784) and a batch of 32 colour images of 224 by 224 pixels has shape (32, 3, 224, 224). The official NumPy[2] documentation calls this attribute shape and uses it in every operation.
Matching shapes is the most common source of bugs when coding. To multiply $\mathbf{W}$ by $\mathbf{x}$, the number of columns of $\mathbf{W}$ must equal the number of rows of $\mathbf{x}$: if $\mathbf{W}$ is (128, 784), then $\mathbf{x}$ must have 784 elements. When a network fails with a dimension error, it is almost always a shape that does not line up.
Rule of thumb: in $\mathbf{W}\mathbf{x}$, the columns of $\mathbf{W}$ must match the length of $\mathbf{x}$. When you hit a dimension error while debugging, always start by printing the shape of each tensor.
Basic operations
These structures work with only a few operations, but they repeat millions of times:
- Element-wise addition: two vectors or two matrices of the same shape are added position by position.
- Scalar product: multiply every element by the same number, as when applying the learning rate $\eta$.
- Dot product: multiply two vectors element by element and sum the result into a single number.
- Matrix-by-vector multiplication: repeat the dot product of each row of the matrix with the vector, which is exactly what a layer does.
The book Mathematics for Machine Learning[3] devotes its first chapters to these operations because they are the basis of everything else. As that text puts it, «linear algebra is the study of vectors and certain rules to manipulate vectors», and those rules are exactly the ones a network applies at every step.
Example in a layer of neurons
Picture a layer with 3 inputs and 2 neurons. The input is a vector $\mathbf{x}$ of 3 values, the weights form a matrix $\mathbf{W}$ of 2 rows and 3 columns, and the bias $\mathbf{b}$ is a vector of 2 values. The computation produces a vector $\mathbf{z}$ of 2 values, one per neuron:
$$\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}, \qquad \mathbf{W} \in \mathbb{R}^{2 \times 3},\ \mathbf{x} \in \mathbb{R}^{3},\ \mathbf{z} \in \mathbb{R}^{2}$$
An activation function $\mathbf{a} = f(\mathbf{z})$ is then applied to each component, and those 2 numbers are the output that passes to the next layer. If you chain several layers with this pattern you get a full network: each layer $^{(l)}$ takes the vector from the previous one, multiplies it by its weight matrix and transforms it. Nothing more complicated than multiplying tables by lists, repeated in cascade.
Frequently asked questions
Is a tensor the same as a multidimensional array?
In deep learning, in practice yes: a tensor is an array of numbers with any number of dimensions. In physics and geometry the term tensor has a stricter meaning that includes transformation rules, but when you program a network it is enough to think of it as a table of numbers with several dimensions.
Why are vectors and matrices used in neural networks?
Because they let you express and compute millions of operations at once. Writing a layer as a matrix-by-vector multiplication lets the graphics card resolve thousands of dot products in parallel, something impossible if we handled each neuron separately.
Do I need to master linear algebra before starting?
You do not need a full course. Understanding what a vector is, what a matrix is, how they multiply and what a tensor’s shape means gives you 90% of what is used day to day. The rest you learn as you go.
Conclusion
Scalars, vectors, matrices and tensors are not four different things but the same idea at 0, 1, 2 and more dimensions. With that staircase in mind, the formula $\mathbf{z} = \mathbf{W}\mathbf{x} + \mathbf{b}$ stops being a puzzle and becomes what it is: multiplying a table of weights by a list of data. The natural next step is to review the roadmap of the mathematics of neural networks and to practise with real shapes.
