adagrad

AdaGrad and RMSProp: Adaptive Learning Rates

AdaGrad and RMSProp are optimisers that give every weight its own learning rate. AdaGrad accumulates the square of all past gradients and divides the step by its root, which slowly fades it out. RMSProp fixes this with a moving average that forgets the past using a decay factor of about 0.9.