Tag
FreeToken is an open-source inference engine, released by UC Berkeley and UT Austin researchers in August 2026, that splits a MoE model's experts across GPU, CPU and system memory according to each machine's measured bandwidth. It serves models from 35B to 753B parameters on consumer hardware.
NVIDIA still dominates frontier-model training in 2026, but inference tells a different story. AMD MI300X/MI325X with mature ROCm, Intel Gaudi 3, Google TPU v6, and AWS Trainium/Inferentia deliver 20 to 50% lower cost per token without sacrificing quality. Here is when to choose each option.
TensorRT-LLM is the NVIDIA inference engine that compiles each model into a binary optimized for the exact GPU and batch size it will serve. It uses hand-written CUDA kernels and native FP8 quantization on H100. Against vLLM it can run 2 to 3 times faster in the best case, at the cost of a 30 to 90 minute build.
