Updated: 2026-07-07

Serving an open LLM in production is not trivial: keeping the GPU saturated, batching requests without blocking, applying quantization without losing quality, and exposing an OpenAI-compatible API means solving several problems at once. Text Generation Inference (TGI)[1] from Hugging Face tries to resolve them with a coherent stack. It is one of the more mature options in the open ecosystem, with a somewhat bumpy licensing past worth knowing before you adopt it.

Versión en español de este artículo

Key takeaways

  • TGI supports continuous batching, FlashAttention v2, tensor parallelism, and quantization (bitsandbytes, GPTQ, AWQ) with minimal configuration.

  • For Hugging Face Hub models, TGI is the lowest-friction path: load by ID, no format conversion.

  • TGI briefly switched to a restrictive licence (HFOIL) between 2023 and early 2024, then reverted to Apache 2.0 in version 2.0: today it is as open as vLLM or llama.cpp[2].

  • For raw throughput on high-end GPUs, vLLM usually wins. For CPU or Apple Silicon, llama.cpp is better.

  • The OpenAI-compatible API (/v1/chat/completions) enables hosted-to-self-hosted transitions without code changes.

What TGI is

TGI is an inference server specialised in text generation, written mostly in Rust (the router) and Python (the worker using PyTorch). It supports the most popular hub models: Llama 2, Mistral, Falcon, StarCoder, BLOOM, CodeLlama, Yi, MPT, Phi, and variants.

Key capabilities:

  • Continuous batching: groups concurrent requests without waiting for the slowest. Maximises GPU utilisation.

  • FlashAttention v2: optimised attention implementation reducing memory and accelerating computation.

  • PagedAttention: KV-cache memory management inspired by vLLM (though vLLM implements it better).

  • Tensor parallelism: split the model across multiple GPUs.

  • Quantization: bitsandbytes (NF4, FP4, INT8), GPTQ, AWQ, EETQ. 4-bit or 8-bit inference with minimal effort.

  • SSE streaming: tokens sent to client as generated.

  • Guided generation: grammars, regex, JSON schema via Outlines.

  • OpenAI-compatible API: /v1/chat/completions as an additional layer.

Minimal deployment

docker run --gpus all --shm-size 1g -p 8080:80 
  -v $PWD/data:/data 
  ghcr.io/huggingface/text-generation-inference:latest 
  --model-id mistralai/Mistral-7B-Instruct-v0.2 
  --quantize bitsandbytes-nf4

After a few minutes loading, the endpoint responds at POST /generate with streamed tokens. On Kubernetes, the official chart covers deployments with GPU operator; pods need nvidia.com/gpu: 1 (or more for tensor parallelism).

NVIDIA logo, manufacturer of the GPUs on which TGI runs in production for large language model inferenceNVIDIA logo, manufacturer of the GPUs on which TGI runs in production for large language model inference

Where TGI shines

TGI is the right option for:

  • Direct Hugging Face Hub models: load by ID, no format conversion.

  • bitsandbytes-quantized models: native NF4, FP4, INT8 support.

  • High-quality SSE streaming: low first-token latency, stable throughput.

  • Hugging Face Inference Endpoints: TGI is the engine behind their managed service.

  • Transformers integration: same tool family, proven compatibility.

If you’re already in the HF ecosystem, TGI is the lowest-friction path.

Where others surpass

  • Raw throughput: vLLM[3] squeezes more tokens/second on high-end GPUs thanks to its well-implemented PagedAttention, and is Apache 2.0.

  • CPU or Apple Silicon: llama.cpp[2] / Ollama[4] are better for GPU-less inference.

  • Exotic models: TGI covers the popular ones; less common or very new models may be unsupported.

  • The 2023 licence scare, now over: Hugging Face released TGI v1.0 under HFOIL[5], a licence that blocked reselling it as a managed service. A fair number of teams got spooked and some migrated to vLLM at the time. But version 2.0 reverted the licence to Apache 2.0[6], so the argument no longer holds: if someone tells you TGI "has a restrictive licence," they are quoting a 2023 headline that is no longer true.

High-impact optimisations

Three small tunings with large effect:

  • --max-batch-prefill-tokens: total token cap in prefill phase (the costliest). Higher means more concurrency but more VRAM.

  • --max-total-tokens: maximum context window per request. Tighter means less memory use.

  • --quantize gptq or --quantize awq: better than bitsandbytes if you have a pre-quantized model.

Measure throughput before and after with locust[7] or vegeta[8] to validate real impact.

OpenAI-compatible API

curl http://localhost:8080/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "model": "tgi",
    "messages": [{"role": "user", "content": "Hello"}],
    "stream": true
  }'

This lets you use LangChain[9], LiteLLM[10], and OpenAI SDKs without code changes. The transition from a hosted model to self-hosted goes from weeks to minutes.

Production operation

Checklist for serious TGI ops:

  • Health checks against /health.

  • Prometheus metrics at /metrics, which expose latency, throughput, and VRAM usage.

  • Limit concurrency at router level to avoid OOM in spikes.

  • Backup the downloaded model: if the hub is unreachable you can’t start.

  • Monitor GPU temperature: above 90°C it degrades and fails.

  • Plan for CUDA and driver updates, since TGI is version-sensitive.

PyTorch logo, the deep learning framework used by TGI's worker for language model inferencePyTorch logo, the deep learning framework used by TGI’s worker for language model inference

Alternatives to consider

  • vLLM[3]: better general throughput, very active community, Apache 2.0.

  • llama.cpp[2] / Ollama[4]: CPU and Apple Silicon, simpler deployment.

  • TensorRT-LLM[11]: performance ceiling on NVIDIA GPUs, but high operational complexity.

  • LMDeploy[12]: very good performance on certain models.

Conclusion

TGI remains a robust, sensible choice for most teams serving open models: support for popular hub models, easy quantization, familiar API, and HF-ecosystem integration. And contrary to what a lot of people still repeat, the licence is no longer a hurdle: since version 2.0 it is Apache 2.0 again. For absolute throughput on top GPUs, vLLM usually wins; for maximum simplicity on CPU, llama.cpp. If you are in the HF ecosystem, TGI remains the lowest-friction option, without the fine print it carried back in 2023.

Sources

  1. Text Generation Inference (TGI)
  2. llama.cpp
  3. vLLM
  4. Ollama
  5. Hugging Face released TGI v1.0 under HFOIL
  6. version 2.0 reverted the licence to Apache 2.0
  7. locust
  8. vegeta
  9. LangChain
  10. LiteLLM
  11. TensorRT-LLM
  12. LMDeploy
  13. Official Text Generation Inference documentation