Discussion - Topic: AI Model Routers
Details
This week's topic: AI Model Routers
Discussion resources to help guide the conversation will be posted below a few days before the meetup.
Zoom link will be added about 5 min before the event starts.
AI model routers are systems that automatically direct each request to the most suitable AI model based on factors such as task complexity, accuracy, speed, cost, privacy, and availability. We’ll discuss how routing works, when it offers advantages over relying on a single model, and the practical challenges involved—including evaluation, fallback strategies, observability, vendor lock-in, and governance. The conversation will focus on real-world architectures, tradeoffs, and whether model routing is mature enough for production use.
Discussion Resources :
Router — Load Balancing By LiteLLM Contributors
https://docs.litellm.ai/docs/routing
Life of a Request By LiteLLM Contributors
https://docs.litellm.ai/docs/proxy/architecture
RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing By Ivan Ortega and UC Berkeley Sky Computing Lab Contributors
https://sky.cs.berkeley.edu/project/routellm/
Model Routing Powered by Wisdom of the Market By OpenRouter Team (August 2026)
https://openrouter.ai/blog/announcements/introducing-the-new-auto-router/
How OpenRouter Model Routing Works By OpenRouter Team (2026)
https://openrouter.ai/blog/insights/model-routing/
Understanding Intelligent Prompt Routing in Amazon Bedrock By Amazon Web Services
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html
Model Routing Is an Evaluation Problem By Brandon Lei, with Vaithiyanathan Sundaram, Michael Wang, Biao Zhang, Braden Holstege, Han Pei, Alex Gonzalez, Helen Rhee, and Imani Parker-Fong (July 2026)
https://www.mercor.space/blog/mercor-model-routing-eval-problem/
Up and to the Left! How Martian Uses Routing to Push the Pareto Frontier By Adam Michael Wood
https://withmartian.com/post/up-and-to-the-left
RouteLLM: Learning to Route LLMs with Preference Data By Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica (ICLR 2025)
https://arxiv.org/abs/2406.18665
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance By Lingjiao Chen, Matei Zaharia, and James Zou
https://arxiv.org/abs/2305.05176