What Is an Activation Function and Why It Is Needed
About this video
Why a non-linear activation function makes deep learning possible.
An activation function is the nonlinear operation each neuron applies to its weighted sum z to produce its output a equals f of z. Without it, stacking layers only chains linear transformations and the whole network collapses into a single one. That nonlinearity is what lets a network learn complex patterns.
Video transcript
- 0:01 Sigmoid, tanh and ReLU bend the straight line.
- 0:04 Without that bend, stacking layers adds up to a single linear layer; with it, the
- 0:09 network learns complex patterns.
More videos
Explainer
The Computational Graph and Automatic Differentiation
The computational graph and automatic differentiation, in action.
Explainer
Backpropagation: The Step-by-Step Derivation
The backpropagation derivation, factor by factor with the chain rule.
Explainer
Backpropagation: The Intuition
The intuition of backpropagation: the error flows backward through the network.
Explainer
Batch, Stochastic and Mini-Batch Gradient Descent
Batch, stochastic and mini-batch: three ways to traverse the data.