Tag
NVIDIA still dominates frontier-model training in 2026, but inference tells a different story. AMD MI300X/MI325X with mature ROCm, Intel Gaudi 3, Google TPU v6, and AWS Trainium/Inferentia deliver 20 to 50% lower cost per token without sacrificing quality. Here is when to choose each option.
FinOps for AI counts different units than classic cloud FinOps: tokens, calls, computed embeddings and GPU time, all of which scale nonlinearly with use. The costliest habit is sending everything to frontier models; 40 to 70 percent of those calls run on mid-tier models with no noticeable quality loss. Uncached RAG and self-recursing agents do the rest.
NVIDIA Blackwell GPUs change large-model training mainly at rack scale. The GB200 NVL72 packs 72 Blackwell GPUs and 36 Grace CPUs into around 120 kW and presents itself to software as one machine holding 13.4 TB of HBM3e. Official MLPerf Training v5.0 numbers show 2.2x on Llama 3.1 405B pretraining against H100.
vLLM remains the reference engine for serving LLMs on GPU in 2025: automatic prefix caching sharply cuts latency for repeated prompts, speculative decoding speeds up large models, and multi-LoRA support lowers the cost of multi-tenant SaaS, though multi-GPU support and non-NVIDIA hardware remain weak points.
TensorRT-LLM is the NVIDIA inference engine that compiles each model into a binary optimized for the exact GPU and batch size it will serve. It uses hand-written CUDA kernels and native FP8 quantization on H100. Against vLLM it can run 2 to 3 times faster in the best case, at the cost of a 30 to 90 minute build.
vLLM serves language models on GPU using PagedAttention and continuous batching, two techniques that multiply throughput compared with a naive server. It exposes an OpenAI-compatible API, so migrating an existing application only requires changing the base URL and deploying the right binary.
Zed is the editor built by Atom's creators, rebuilt in Rust with a native GPU-rendering UI framework and no Electron. It delivers ~8 ms latency, real-time collaboration with integrated voice and shared cursors, and an open license (GPL v3 + Apache 2.0). A serious VS Code alternative for pairs and small teams.
Text Generation Inference (TGI) is the Hugging Face stack for serving open LLMs in production: continuous batching, 4-bit and 8-bit quantization, streaming, and an OpenAI-compatible API. After a brief restrictive-licence episode in 2023, it returned to Apache 2.0 in version 2.0.
