Architecture
Inference routers: choosing a model based on the request
An inference router decides which model answers each incoming request, weighing cost, latency and how hard the request actually is. Well-built inference routers cut total token spend by 30 to 70 percent with no quality loss the user can perceive. Four patterns cover most cases: length, task type, an auxiliary classifier, and learned routing.
