An AI Router Is Three Things. Adopt Them One at a Time.

September 18, 2026 · 19 min read

Software Engineer

A glowing railway track splits at a switch in a desert at dusk, its branches carrying beads of light toward three distant cities: one bright and welcoming, one lost in fog, one under a storm

An artificial intelligence (AI) router sits between your app and the model providers and decides, per request, which model answers. "Router" names three different jobs — a gateway, a fallback chain and a smart router — and each hands over more of a decision that used to be yours.

note

This is a snapshot of a fast-moving market as of September 2026. Fees, features and defaults change often — check each vendor's current docs before you decide anything on the numbers below.

The gist

Pick by size: the smaller you are, the more of the router you can rent; the bigger, the more of it you must own.

Solo developer: a hosted gateway, or nothing. One provider needs no router; for many models, OpenRouter's fee is small change.
Team: self-host a gateway with a fallback chain. Per-team keys and budgets are the payoff; pin its version.
Company: keep it inside your perimeter — self-hosted, or your cloud's built-in router.
Pin whatever you evaluate. Same model, same provider, fallbacks off.
Add smart routing last, and only with evals showing your prompts are mostly easy.

The mental model: three jobs under one name

A router is a reverse proxy for large language model (LLM) calls: your app talks to one endpoint, and that endpoint talks to the providers. What makes it more than a proxy is that it holds opinions about where each request should go.

"Router" is used for three different jobs, stacked on top of each other, and most confusion about these tools comes from not separating them:

The gateway — one application programming interface (API), one key, many providers. It translates your request into whatever shape each provider expects. Nothing is decided; it's plumbing.
The fallback chain — retries, failover and load balancing. When a provider errors or is overloaded, the request goes to the next one. The router decides who serves you, but only when something fails.
The smart router — picks a model for each prompt, usually sending easy prompts to a cheap model and hard ones to an expensive one. The router decides who serves you every time.

Each layer up hands the router more of a decision that used to be yours, and the price rises with it. The gateway is nearly all upside. The fallback chain is a fair trade. The smart router is something you have to earn. They can be adopted one at a time, and should be.

What happens to a request

Before weighing the layers, it helps to see one request go through all three. The app asks for a model — by name, or by handing the choice over entirely — and the router authenticates the caller, checks its budget, picks a target, and tries again elsewhere if the first target fails.

The "pick" step is where products differ. OpenRouter, a hosted router, by default first sets aside providers that had an outage in the last 30 seconds, then load-balances across the rest weighted by the inverse square of their price — a provider at a third of the price is nine times more likely to get the request — and keeps the others as fallbacks^[1]. Its Auto Router goes a layer further and chooses the model too, reporting the choice back in the response's model field^[2].

Hold on to that last detail. The response telling you what actually served it is the single most important feature in this whole category. Everything a router can quietly get wrong is only catchable if you know who answered.

The gateway: nearly all upside

The first layer decides nothing, which is exactly why it's the easy one to say yes to. What it gives you:

One API, one key, one bill. LiteLLM, the best-known open-source gateway, exposes a single OpenAI-compatible endpoint in front of more than a hundred providers^[3]. Your code learns one request shape and one error shape, and finance gets one invoice.
Switching models becomes configuration. Trying a new model, or leaving a provider that raised prices, is a change to a config file rather than to a software development kit (SDK) integration. That optionality is worth more every month the model market keeps moving.
Central control. Virtual keys per team or per feature, spend budgets and rate limits live in one place^[4]. Cloudflare's AI Gateway adds response caching on top of the same set^[5]. Without a gateway, each of these gets reinvented per service, or not at all.
One place to look. Tokens, cost and latency per team and per feature, across every provider, without stitching dashboards together.

What it costs is the honest price of putting one more network hop in front of everything:

A new single point of failure. When the gateway is down, every provider is down for you at once. A hosted gateway makes that someone else's uptime; a self-hosted one makes it yours.
A fee, or an ops bill. OpenRouter passes model prices through without markup but charges 5.5% on credit purchases, and 5% on bring-your-own-key (BYOK) usage above a monthly allowance^[6]. Vercel's AI Gateway charges no markup at all, BYOK included^[7]. Self-hosting LiteLLM costs nothing in fees and something in servers, upgrades and on-call.
A lowest-common-denominator API. A unified request shape can only carry what all providers share. Anything a provider does differently — caching controls, tool-calling quirks, reasoning settings — either leaks through as a provider-specific escape hatch or is lost.
It holds every key you own. The gateway is the one process with credentials for all your providers, which makes it the best target in your stack. In March 2026 two releases of LiteLLM on the Python Package Index (PyPI), 1.82.7 and 1.82.8, were published by an attacker and harvested environment variables, Secure Shell (SSH) keys and cloud credentials from whoever installed them unpinned during the hours they were live; users of the official Docker image, which pins its dependencies, were not affected^[8].

None of these costs changes what your app says: a gateway can fail loudly, but it can't make your answers quietly different. That's the line the next two layers cross.

The fallback chain: a fair trade

The second layer gives you one thing, and it's a good one: failover you don't have to write. Every provider has bad afternoons. A fallback chain turns "the provider is down" from an incident into a log line — and it's the kind of code that is tedious to write well and easy to get subtly wrong.

In exchange, the router now sometimes chooses who serves you, and "sometimes" has three consequences.

It breaks the provider's biggest discount. Anthropic's prompt cache bills repeated prompt prefixes at a tenth of the normal input price, but a hit requires a 100% identical prefix, the entry expires after five minutes by default, and caches are never shared between organisations^[9]. A router that spreads consecutive requests across providers lands each one on a cold cache, and the discount silently disappears. OpenRouter counters this with sticky routing — it remembers which provider served a cached conversation and keeps sending it there — but when that provider fails and the fallback kicks in, the next request pays full price again^[10].

The same model name is not the same service. Open-weight models are hosted by many providers, and a fallback chain treats them as interchangeable. They aren't. Providers run different inference stacks and different quantisations — the precision the model's weights are stored at, traded against speed and cost. OpenRouter itself said so when it launched its Exacto endpoints: in theory "the same model weights (with the same quantization) should yield the same results", but in practice "differences emerge", and it measured tool-calling accuracy varying significantly between providers of one model^[11]. A 2026 measurement study of hosted open-weight APIs reached the general form of the conclusion: these are "heterogeneous services, not static catalog entries"^[12]. With price-weighted routing, the cheapest provider gets most of your traffic, and which provider is cheapest is not something you control.

Your prompts go to parties you didn't pick. A hosted router is a third party reading your traffic, and behind it sit providers chosen at request time, each with its own retention policy. The defaults are often reasonable — OpenRouter doesn't log prompts or completions unless you opt in^[6] — but "a provider selected by price" is a hard sentence to put in a data-processing agreement. The controls exist: OpenRouter can restrict routing to providers that don't store data, or to zero-data-retention endpoints only^[1]. They are just not on by default.

The trade is fair as long as the chain is one you wrote: a short, explicit list of providers you've vetted, rather than whoever is cheapest today.

The smart router: earn it first

The third layer is the one the name promises, and its benefit is real — on benchmarks. RouteLLM, the reference open-source smart router, trained on human preference data to send easy prompts to a weak model and hard ones to GPT-4. It kept 95% of GPT-4's quality while cutting cost by over 85% on a chat-quality benchmark, 45% on a general-knowledge one and 35% on school maths problems^[13].

That spread — 85% on one benchmark, 35% on another — is the catch. How much a smart router saves depends entirely on how many of your prompts are easy, and nobody's benchmark can tell you that. Only your own evaluations can, and the smart router is also what makes those evaluations hard.

An evaluation run is only meaningful if production runs the thing you evaluated. Behind a router that chooses, it doesn't: the eval measured whatever mix happened to serve it that day. Even one layer down this already goes wrong — an audit of AI-safety research codebases found 31 of 32 using OpenRouter without pinning a provider, leaving their results open to exactly this contamination^[14]. A smart router multiplies it, because now the model varies too, and the selection logic is itself a moving part. OpenRouter's Auto Router ranks models by aggregate market spend over a trailing seven-day window^[2] — so which model answers your prompt can change next week with no deploy, no config change and no changelog entry on your side.

Debugging inherits the same problem. When a user reports a bad answer, the first question used to be "what was the prompt?". Behind a smart router there's one before it: what produced this? If you didn't record the model field and the provider from each response, that question has no answer, and the bug is unreproducible by construction.

So the smart router is the one layer with an entry fee: evals that run through it, on a schedule, and a log of what served every response. Without those, you aren't saving money — you're not measuring what you lost.

Pinning: the cure that switches the router off

Every hidden cost in the last two sections has the same fix: take the decision back. This is what a pinned OpenRouter request looks like — one provider, no silent fallback, a stated precision, no data-retaining hosts:

{
  "model": "moonshotai/kimi-k2",
  "provider": {
    "order": ["deepinfra"],
    "allow_fallbacks": false,
    "quantizations": ["fp8"],
    "data_collection": "deny"
  }
}

Notice what that did. It switched off layers two and three and kept only the gateway. Pinning is the cure for everything a router does silently, and pinning is also the act of not using the router as a router. That tension doesn't resolve; you choose where on it to stand, per workload — pinned for anything you evaluate or audit, looser where an occasional odd answer is cheap.

Which one do you need?

Work down from the least machinery that solves your actual problem.

The names you'll meet sort onto the same three layers:

Gateways with fallback chains. OpenRouter, LiteLLM, Vercel AI Gateway and Cloudflare AI Gateway, plus two open-source gateways that come with a hosted option: Portkey, which adds caching and guardrails on top of fallbacks and load balancing^[15], and Helicone's gateway, written in Rust and tied to its observability product^[16]. If your company already runs Kong as its API gateway, its AI Proxy plugins do the same job, including routing by the semantic similarity of the prompt^[17].
Commercial smart routers. Not Diamond predicts, per prompt, which model will give the best answer at the lowest cost, with a pre-trained router or one trained on your own data^[18]. Martian sells the same promise^[19]. OpenRouter's Auto Router and the open-source RouteLLM sit here too.
Smart routers built into a cloud. Amazon Bedrock's Intelligent Prompt Routing chooses between models of one family behind a single endpoint^[20], and Microsoft Foundry's model router is itself a trained model that you deploy like any other, with balanced, cost and quality modes^[21]. Staying inside one cloud removes the extra third party, and narrows the pool the router can choose from.

For the gateway itself, the main options differ on two things that matter — what they charge and whether a third party sits in your request path:

Option	What it is	Platform fee	Third party in the path
OpenRouter	Hosted gateway, fallbacks, and a smart Auto Router	5.5% on credits	Yes
Vercel AI Gateway	Hosted gateway with fallbacks and budgets	None	Yes
Cloudflare AI Gateway	Hosted gateway with caching, rate limits, fallbacks	Available on all Cloudflare plans	Yes
Portkey	Open-source gateway with guardrails; hosted or self-run	None if self-hosted	Your choice
Helicone AI Gateway	Open-source gateway with observability; hosted or self-run	None if self-hosted	Your choice
LiteLLM	Open-source gateway you run yourself	None — you pay in ops	No
RouteLLM	Open-source smart-router framework, no gateway	None	No

Who you are narrows it further:

You are	Pick	Why	Skip
A solo developer	No router with one provider. For many models: OpenRouter, or Vercel AI Gateway if you already deploy there	One key, every model, nothing to run; at hobby spend a 5.5% fee is small change, and Vercel charges none	Self-hosting, smart routing
A team	A self-hosted gateway — LiteLLM or Portkey — with a fallback chain; a hosted one only if prompts may leave	Per-team virtual keys, budgets and rate limits are the real payoff; run the pinned Docker image, never an unpinned install	Smart routing, until you have evals
A company	A gateway inside your perimeter: Kong if you already run it, else self-hosted LiteLLM or Portkey — or your cloud's built-in router (Bedrock, Microsoft Foundry)	Keeps the list of parties that see your prompts at one; central audit and budgets; zero-data-retention routing where a hosted hop is unavoidable	An unpinned hosted router on anything regulated

The smaller you are, the more of the router you can rent; the bigger you are, the more of it you have to own.

Whichever you pick, four habits keep it on the good side of the ledger:

Pin anything you evaluate. Same model, same provider, fallbacks off — for the eval run and for the production path it vouches for.
Log what served every response. Model and provider, next to the prompt. It's one field, and it's the difference between a bug and a ghost story.
Run your evals through the router, not around it. If production goes through smart routing, the thing to evaluate is the router plus its pool, on a schedule — not one model on one day.
Keep a direct path. One code path that can call your main provider without the router, behind a flag. It's your failover for the failover.

The short version

A "router" is three jobs: a gateway (one API), a fallback chain (decides on failure), and a smart router (decides every time). Adopt them separately, in that order.
The gateway is nearly all upside — one key and bill, budgets, switching models by config — at the price of a new hop that can fail and that holds every key. It can break loudly, but it can't change your answers.
The fallback chain buys failover and pays in silent variation: cold prompt caches, the same model name served differently, prompts reaching providers you didn't choose. Write the chain yourself.
The smart router saves real money only if your prompts are mostly easy, and only your own evals — run through the router — can show that.
Let a router decide only what you're able to check afterwards. Pin what you evaluate, log what served each response, and keep a way around it.

The gist​

The mental model: three jobs under one name​

What happens to a request​

The gateway: nearly all upside​

The fallback chain: a fair trade​

The smart router: earn it first​

Pinning: the cure that switches the router off​

Which one do you need?​

The short version​

References​