TensorRT-LLM: Extreme Acceleration on NVIDIA GPUs for LLMs

TensorRT-LLM is the NVIDIA inference engine that compiles each model into a binary optimized for the exact GPU and batch size it will serve. It uses hand-written CUDA kernels and native FP8 quantization on H100. Against vLLM it can run 2 to 3 times faster in the best case, at the cost of a 30 to 90 minute build.

November 4, 2024 5 min 226

Architecture

vLLM: Serving LLMs in Production with Very High Throughput

vLLM serves language models on GPU using PagedAttention and continuous batching, two techniques that multiply throughput compared with a naive server. It exposes an OpenAI-compatible API, so migrating an existing application only requires changing the base URL and deploying the right binary.

October 5, 2024 7 min 304 4.5