The Linear Activation Function

The linear activation function is f(x) = ax + b and, in the form networks use, the identity f(x) = x: the neuron outputs its weighted sum as it is. It has a constant derivative and an unbounded range, which is why it is the standard activation for the output layer of a regression. In hidden layers it adds nothing, because two stacked linear layers equal a single one. This guide explains both facts with the formula, a check in code and the cases where it does belong. The same explanation is available in Spanish.

Key takeaways

• The linear activation function is f(x) = ax + b; with a = 1 and b = 0 it is the identity f(x) = x, the form PyTorch and Keras use.
• Its derivative is the constant a: the gradient flowing through the layer neither saturates nor vanishes.
• Composing linear layers gives another linear layer: W₂(W₁x + b₁) + b₂ = W'x + b', so depth adds no capacity.
• It is the right activation for a regression output, where the target can be any real number.
• It also appears inside modern architectures: the query, key and value projections of a transformer are linear layers with no activation.

What is the linear activation function?

The linear function is the activation function that transforms nothing. A neuron first computes the weighted sum z = Wx + b (covered in weights, biases and the weighted sum) and then applies a = f(z). With the linear function, a is z scaled by a slope and shifted by a constant.

In practice the slope and the shift are redundant, because the layer’s own weights and bias already absorb them. That is why libraries implement the identity: Keras calls it linear and documents it as an identity function, and PyTorch ships the module torch.nn.Identity^[1], which returns its input unchanged. When a Keras dense layer is created without the activation argument, this is the function it applies.

Formula, derivative and range

The general definition has two parameters, the slope a and the intercept b:

$$f(x) = ax + b$$

Its derivative is the same at every point:

$$f'(x) = a$$

Its three useful properties follow from that. The domain and the range are all real numbers, so the output does not saturate the way the sigmoid does. The gradient coming back through backpropagation is always multiplied by a (by 1 for the identity), without the vanishing that bounded activations suffer. And it is continuous and differentiable everywhere, without the kink that ReLU has at the origin.

Why does it fail in hidden layers?

A network whose hidden layers use a linear activation is equivalent to a single linear layer, whatever its depth. With two layers, the second receives the output of the first and the result rearranges like this:

$$W_2(W_1 x + b_1) + b_2 = (W_2 W_1)\,x + (W_2 b_1 + b_2) = W’x + b’$$

This Python snippet with NumPy checks it with random matrices. It builds a two-layer network with a linear activation, computes the equivalent single layer and compares the outputs; then it repeats the comparison with ReLU between the layers:

import numpy as np

rng = np.random.default_rng(0)
W1, b1 = rng.normal(size=(4, 3)), rng.normal(size=4)
W2, b2 = rng.normal(size=(2, 4)), rng.normal(size=2)
x = rng.normal(size=3)

# Two layers with a linear (identity) activation
y = W2 @ (W1 @ x + b1) + b2

# One equivalent layer
W, b = W2 @ W1, W2 @ b1 + b2
print(np.allclose(y, W @ x + b))  # True

# With ReLU between the layers that shortcut is gone
y_relu = W2 @ np.maximum(0, W1 @ x + b1) + b2
print(np.allclose(y_relu, W @ x + b))  # False

We ran it on 27 October 2026 in a python:3.13-slim container (Python 3.13.18 and NumPy 2.5.4) and the output was True and then False. The equivalent layer reproduces the linear network exactly, but not the network with ReLU.

When should you use the linear function?

The linear function has three legitimate uses, and all three sit outside general-purpose hidden layers:

• Regression output: if the network predicts a price, a temperature or a demand figure, the last layer must be linear. A sigmoid would confine the output to (0, 1) and a ReLU would rule out negative values.
• Projections inside a transformer: the query, key and value matrices of the attention mechanism described in Attention Is All You Need^[2] are linear maps with no activation; the nonlinearity comes from the attention softmax and the feed-forward network.
• Linear bottlenecks: MobileNetV2 deliberately removes the activation from the narrow layer of each block, because its authors found that removing the nonlinearity in those narrow layers preserves their representational power (Sandler et al., 2018^[3]).

For a classification output the choice is different. Binary classification uses the sigmoid, which returns a probability between 0 and 1, and multiclass classification uses the Softmax function, which spreads the probability across all classes.

How the linear function relates to ReLU

ReLU is a piecewise linear function: it is 0 for x ≤ 0 and the identity x for x > 0. That linear half gives it a gradient of exactly 1 in the active region, the same advantage the linear function has over saturation. The fold at the origin is the nonlinearity the identity lacks, and it is enough for stacked layers to add capacity.

The Softplus function takes the same idea one step further: it rounds ReLU’s fold with ln(1 + eˣ) and hugs the line y = x for large inputs. The three functions share the linear behaviour in the positive region and differ in how they treat negative values.

How to write it in PyTorch and Keras

In both libraries you get the linear activation without writing any function. In PyTorch, an nn.Linear layer with no activation module after it already produces a linear output; nn.Identity is there for when an architecture requires a module in that slot. This regression model ends like this:

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(8, 32),
    nn.ReLU(),
    nn.Linear(32, 1),  # linear output: any real number
)

In Keras, layers.Dense(1) without the activation argument applies the identity, the same as layers.Dense(1, activation="linear"). The loss that goes with a linear regression output is the mean squared error, or the Huber loss^[4] if the data has outliers.

The linear output and outliers

A linear output does not saturate, so every outlier in the training set pulls the prediction as hard as its error. With the mean squared error that pull grows with the square of the error, and a few extreme points can bias the model. There are three standard mitigations:

• Change the loss: the mean absolute error or the Huber loss, which is quadratic for small errors and linear for large ones.
• Normalise the target: train on a standardised output variable and undo the transformation when predicting.
• Clean the data: clip or drop outliers that come from measurement errors, not the ones that are real cases.

Frequently asked questions

Why is the linear function not used in hidden layers?

Because composing linear maps gives another linear map. A network with a linear activation in every layer is equivalent to a single layer W'x + b', whatever its depth. Learning nonlinear relationships needs a nonlinear activation, such as ReLU, the sigmoid or the hyperbolic tangent, between the hidden layers.

What is the difference between the linear function and the identity function?

The identity is the special case of the linear function with a = 1 and b = 0, that is, f(x) = x. Libraries implement the identity because the layer’s weights and bias already supply the slope and the shift. Keras calls this activation linear and PyTorch ships it as nn.Identity.

Does the linear function suffer from vanishing gradients?

No. Its derivative is constant, so the gradient passing through it is always multiplied by the same value, 1 for the identity. That advantage does not make up for its inability to model nonlinear relationships in hidden layers, which is why ReLU, which keeps a derivative of 1 in its positive region, takes that place.

Conclusion

The linear activation function has a precise place in a neural network: the output layer of a regression, plus the projections and bottlenecks where the architecture wants a transformation without distortion. In general-purpose hidden layers it does not work, because any number of linear layers reduces to a single one. You can place it alongside the other activations within the mathematics behind neural networks roadmap.

Sources: [1] Goodfellow, Bengio and Courville, Deep Learning, chapter 6^[5], [2] torch.nn.Identity, PyTorch documentation^[1], [3] keras.activations.linear, Keras documentation^[6], [4] Attention Is All You Need (Vaswani et al., 2017)^[2], [5] MobileNetV2: Inverted Residuals and Linear Bottlenecks (Sandler et al., 2018)^[3].

Sources

1. torch.nn.Identity
2. Attention Is All You Need
3. Sandler et al., 2018
4. Huber loss
5. Goodfellow, Bengio and Courville, Deep Learning, chapter 6
6. keras.activations.linear, Keras documentation

Route: The Neuron and Activation Functions