AI Engineering/Phase 13/Lesson 21

LLM Routing Layer — LiteLLM, OpenRouter, Portkey

Provider lock-in is expensive. Different tool-calling workloads suit different models. Routing gateways give one API surface, retries, failover, cost tracking, and guardrails. Three archetypes dominate 2026: LiteLLM (open-source self-hosted), OpenRouter (managed SaaS), Portkey (production-grade, open-sourced in March 2026). This lesson names the decision criteria and walks a stdlib routing gateway.

Learn 45 min Python (stdlib, routing + failover + cost tracker)

Prerequisites: Phase 13 · 02 (function calling), Phase 13 · 17 (gateways)

#Learning Objectives

Distinguish self-hosted, managed, and production-grade routing options.
Implement a fallback chain that retries on provider failures in a defined priority order.
Track per-request cost and token usage across providers.
Decide between LiteLLM, OpenRouter, and Portkey for a given production constraint.

Full Access lesson

Keep going. The rest of this lesson is one step away.

Unlock all 503 lessons, video, code, quizzes and every future update. Pay in your own currency with Paystack.

See plans Sign in

The text and code of this lesson are based on AI Engineering from Scratch by Rohit Ghumare, MIT licensed.