Artificial Intelligence
TensorRT-LLM: Extreme Acceleration on NVIDIA GPUs for LLMs
TensorRT-LLM is the NVIDIA inference engine that compiles each model into a binary optimized for the exact GPU and batch size it will serve. It uses hand-written CUDA kernels and native FP8 quantization on H100. Against vLLM it can run 2 to 3 times faster in the best case, at the cost of a 30 to 90 minute build.