colibri

What Colibri is and how it runs giant MoE models by streaming experts from disk

Colibri is an Apache 2.0 inference engine written in C that runs Mixture-of-Experts models of up to 2.8 trillion parameters without loading them whole: it keeps the dense part in RAM and reads from the SSD only the experts the router picks for each token. That is why its speed depends on the disk, the cache and free RAM.