Tag
Second-order methods use the Hessian, the matrix of second derivatives of the loss, to orient each step better than the gradient alone. Newton's method updates the weights by subtracting the inverse Hessian times the gradient, but inverting it is far too costly in large networks, so practitioners fall back on quasi-Newton approximations.
Nesterov accelerated gradient improves classic momentum by evaluating the gradient at a look-ahead point, the one inertia is about to reach, instead of the current point. That anticipation step corrects the trajectory before overshooting and reaches O(1/k²) convergence on smooth convex problems, the fastest a first-order method can achieve.
Momentum is an improvement to gradient descent that accumulates a velocity from past gradients instead of looking only at the current one. With a coefficient β around 0.9, that inertia smooths the zigzag of plain SGD, pushes through long narrow valleys and makes training converge noticeably faster than gradient descent alone.
In large neural networks the loss surface almost never traps training in a bad local minimum. The real obstacle is saddle points, far more common in high dimensions. This article explains local versus global minima, why stochastic gradient descent slips past them, and what role convexity plays in the whole picture of optimisation.
Gradient descent is the algorithm that trains almost every neural network. It computes the slope of the loss function with respect to each weight and takes a small step in the opposite direction, controlled by the learning rate, until it reaches a minimum where the error stops falling and the model has converged.
TensorRT-LLM is the NVIDIA inference engine that compiles each model into a binary optimized for the exact GPU and batch size it will serve. It uses hand-written CUDA kernels and native FP8 quantization on H100. Against vLLM it can run 2 to 3 times faster in the best case, at the cost of a 30 to 90 minute build.
