Categories

Tag

gradient-descent

View Cards
Mode Paginated
Per page Per batch 48
Clear filters
Technology

The Learning Rate in Training

The learning rate is the hyperparameter that sets the size of each step when adjusting the weights during training. Too high a value makes the loss diverge; too low a value makes learning painfully slow. Typical values range from 0.001 with Adam to 0.1 with classic gradient descent.

4 min
Technology

The Exploding Gradient Problem

The exploding gradient problem happens when the gradient norm grows without control during backpropagation, especially in deep and recurrent networks with large weights. Training destabilises and the loss turns into NaN. Gradient clipping, together with good initialisation and normalisation, is the standard fix used today.

3 min