NVIDIA Blackwell GPUs: what changes for training
Updated: 2026-07-07
NVIDIA’s Blackwell architecture was unveiled at GTC 2024, and the first commercial systems started reaching select hyperscalers in late 2024. By September 2025 there are enough real deployments, enough public measurements, and enough operational experience to evaluate what actually changes for large-model training. This review focuses on GB200 NVL72, where Blackwell’s logic is expressed fully.
Key takeaways
-
NVIDIA has shifted from designing GPUs to designing racks: the GB200 NVL72 integrates 72 Blackwell GPUs and 36 Grace CPUs in ~120 kW, presenting to software as a single machine with 13.4 TB of HBM3e.
-
Official MLPerf Training v5.0[1] results show a 2.2x speedup pretraining Llama 3.1 405B and 2.5x on Llama 2 70B LoRA fine-tuning versus H100, not the synthetic 4x NVIDIA advertises, but still a solid gain.
-
Native FP4 is the major inference enabler: a model in FP4 occupies a quarter of the space and uses a quarter of the memory bandwidth, though not every model quantizes equally well to 4 bits.
-
A GB200 NVL72 rack costs around $3 million[2], with lead times that have run 6 to 12 months[3]; practical access in 2025 is via cloud (CoreWeave, Oracle, Azure, Google Cloud) at $10-27/GPU-hour depending on provider[4].
-
For most companies training their own models, Blackwell only pays off if training time is the dominant bottleneck.
The core idea: the rack as unit
The most distinctive thing about Blackwell is not the GPU itself, even though the leap over Hopper is substantial. It is that NVIDIA has stopped designing GPUs and started designing racks. GB200 NVL72[5] integrates 72 Blackwell GPUs and 36 Grace CPUs in a single roughly 120-kilowatt cabinet, interconnected by fifth-generation NVLink at 1.8 terabytes per second. To the software, the rack behaves as a single machine with 13.4 terabytes of unified HBM3e memory.
This is a significant conceptual shift. With Hopper and earlier generations, model parallelism required partitioning the model across GPUs and explicitly managing inter-GPU communication through NCCL. With GB200 NVL72, 72 GPUs can access each other’s memory as if local, albeit with higher latency. This simplifies training patterns where communication between partitions is intense, such as tensor parallelism or mixture-of-experts models.
What the industry is measuring for real performance
NVIDIA’s synthetic numbers point to 4x training throughput over H100 at the same precision, and up to 30x inference speedup with mixed FP4. In practice, audited results are more moderate, though still significant. The official MLPerf Training v5.0[6] round, with 201 results from 20 submitting organizations, backs up the figures above: 2.2x pretraining Llama 3.1 405B and 2.5x fine-tuning Llama 2 70B with LoRA, per GPU at the same power budget.
The gap between the theoretical 4x and the audited 2.2x-2.5x is interesting. Part comes from synthetic benchmarks assuming FP8 or FP4 precision, while real training still uses a lot of BF16 for stability. Part comes from NVIDIA’s Blackwell software stack, in particular the newest cuDNN and TransformerEngine releases, still maturing.
The other relevant number is power. A Blackwell B200 draws 1,000 watts versus 700 for H100[7]. A full GB200 NVL72 rack sits at 120 kilowatts, about 10x a traditional general-purpose server rack. This forces direct liquid cooling at the chip, something that was niche in 2023 and has become the default for AI data centers by 2025.
NVIDIA GB200 NVL72 rack at COMPUTEX 2024, the system that shifts the purchase unit from individual GPU to full cabinet with liquid cooling and 120 kW density, redefining AI data center requirements (Image: 极客湾Geekerwan, CC BY 3.0, via Wikimedia Commons)
FP4 and quantization
One of Blackwell’s technical novelties is native FP4 support, a 4-bit-per-element precision absent in Hopper. FP4 is not useful for training, where gradients need more precision to converge, but it is very useful for inference on large models. A model trained in BF16 and quantized to FP4 occupies a quarter of the space, uses a quarter of the memory bandwidth, and runs much faster if hardware supports it natively.
The catch is that not every model quantizes equally well at 4 bits. NVIDIA has shown that its FP4 quantization recipe[8], applied through the TensorRT Model Optimizer library, can meet MLPerf’s strict accuracy target on models like Llama 2 70B without retraining. But that depends on careful per-model calibration; for mixture-of-experts architectures or tasks that are very sensitive to numerical precision, the quality hit tends to be more noticeable. The current recommendation is FP8 for high-quality inference and FP4 reserved for cases where cost matters more than the last points of quality.
This has a practical implication: Blackwell is more attractive to operators with massive inference workloads than to operators who only train. Hyperscalers have both workloads and can amortize the hardware across them.
Software: still evolving
The software ecosystem has two levels. At the low level, CUDA 12.8 was the release that added official Blackwell support[9], with cuDNN and NCCL updated alongside it. At the high level, popular frameworks lagged: PyTorch did not ship official support until version 2.7[10], released in April 2025 with CUDA 12.8 wheels; versions 2.5 and 2.6 only covered Hopper. JAX moved earlier: NVIDIA’s 25.01 container, built on jaxlib 0.4.38, already supported Blackwell as of January 2025.
By September 2025 the ecosystem has been mature for standard model training for several months, though edge cases remain. Serious teams keep a parallel Hopper environment for cross-checks.
Price, availability, and alternatives
In September 2025, a GB200 NVL72 rack still costs around $3 million for a direct buyer, with lead times running several months except for preferred large customers. This means that during 2025 and much of 2026, Blackwell access is essentially via cloud: CoreWeave, Oracle, Azure, and Google Cloud offer instances at a per-GPU-hour price that varies considerably by provider and commitment, as noted in the key takeaways above.
Real alternatives are few. AMD Instinct MI300X has gained share in 2025 with strong performance and a better memory-per-dollar ratio, though the ROCm software ecosystem still trails CUDA. Google TPU v5p remains competitive but is cloud-only within Google Cloud. Intel Gaudi 3 has fallen to a distant third. For a broader look at these competitors, see NVIDIA alternatives in 2026.
When it pays off
For most companies training their own models, Blackwell only makes sense if training time is the dominant bottleneck. For inference, it depends on volume. At low volume, Hopper remains cheaper per query because Blackwell is overprovisioned. At high volume, especially with FP4 quantization, Blackwell can cut cost per query by a factor of 3 to 5.
My take
Blackwell marks a phase change in AI infrastructure worth understanding even if you never touch one of these racks directly. When NVIDIA designs the product as a rack rather than a GPU, it is nudging the industry toward a model where the purchase and operations unit is the full cabinet, with liquid cooling and 120 kilowatts of density.
For application builders, the relevant part is that the largest models will be available faster and cheaper via API. Frontier training cost keeps growing, but unit query cost keeps falling. That favors strategies consuming third-party models through APIs over strategies training bespoke models. Only cases where the model itself is the competitive edge, or where data cannot leave the organization, justify training.
For operators of traditional infrastructure, Blackwell is an invitation to think about which parts of the data-center stack still belong to the general-purpose future and which belong to specialized AI workloads. Over the next five years data centers will likely split into two radically different profiles, and mixing them in the same building will stop making sense.
Also available in Spanish: GPUs Blackwell de NVIDIA: lo que cambia en el entrenamiento.
