training

The Learning Rate in Training

The learning rate is the hyperparameter that sets the size of each step when adjusting the weights during training. Too high a value makes the loss diverge; too low a value makes learning painfully slow. Typical values range from 0.001 with Adam to 0.1 with classic gradient descent.