03Cloud/Commercial

Model router

One model id: classify the task, apply a cost tier, route to measured evidence, fall back to your baseline.

Set model to auto and the gateway picks the model for each request: it classifies the task, applies a cost tier, and routes to the cheapest model with evidence it can hold quality there. Every decision comes back in response headers, and a request that routing cannot answer falls back to your baseline model instead of failing.

Model id: auto
Cost tiers: low · medium · high · xhigh · max
Tier override: auto:<tier> in the model field, or a cost_tier body field
When routing cannot run: The request goes to your project's baseline model.
Receipts: Response headers on every routed request.

One model id#

Send your provider's native request through the gateway with auto as the model. Nothing else about the request changes.

terminal

curl https://gateway.caveman.so/openai/v1/chat/completions \
  -H "authorization: Bearer cave_live_…" \
  -H "content-type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"…"}]}'

typescript

const completion = await client.chat.completions.create({
  model: "auto",
  messages: [{ role: "user", content: "…" }],
});

cave-auto is an alias for auto and behaves identically.

A request that names a real model is never rewritten. Routing only ever acts on requests that ask for it by sending auto.

Cost tiers#

A tier is a cost band, not a quality promise. There are five, in ascending spend: low, medium, high, xhigh, max.

By default you do not set one. The router reads each prompt's complexity and weights cost against quality accordingly, so a one-line question and a long refactor in the same session are spent on differently. To pin a band yourself, either suffix the model id or send a body field:

json

{ "model": "auto:low", "messages": [] }

json

{ "model": "auto", "cost_tier": "high", "messages": [] }

If both are present, the model suffix wins. An unknown tier value is rejected with a client error rather than forwarded upstream as a garbage model name.

A pinned tier narrows the public-matrix rankings. When the decision is driven by your project's own graded outcomes instead, those rankings are not price-binned: the pin does not narrow them, and the tier header stays absent so the receipt never claims it did.

Pinning can also change what the evidence rests on. Bare auto reads every tier's cells and prefers direct evidence wherever it finds it; a pinned tier reads one cell, so a task measured directly in a pricier band may answer a cheap pin from the fallback table instead. That is what asking for the cheap band means.

How a request is routed#

model: auto · your request, unchanged otherwise

↓

1 · Classify. Task type (one of about thirty, like agent:multi_step_planning) and a complexity score that sets the tier, unless you pinned one.

↓

2 · Look up evidence. Your project's own graded outcomes for this task type when there are enough of them, otherwise the public benchmark matrix.

↓

3 · Apply your policy. Allowed models, denylist, provider health, and capacity ceilings only ever narrow the pool.

↓

4 · Route with fallbacks. The top surviving model gets the request; the rest are ordered fallbacks.

↓ anything missing at any step

Baseline. The request goes to your configured baseline model. It never fails because routing was unavailable.

The pipeline narrows, then routes. Every exit lands on a model you approved.

The evidence in step 2 has an order. Once your project has graded outcomes for a task type, roughly twenty or more covering at least two candidate models, the router ranks by what worked on your traffic. Before that, it ranks by the public matrix: measured quality per dollar from published benchmark results, refreshed weekly. Your own evidence always outranks the public table, because it is the only table that measured your workload.

Receipts#

Every routed response carries the decision:

Header	Meaning
`x-cave-router-version`	Which router answered.
`x-cave-route-from`	The model string you sent, for example `auto:high`.
`x-cave-route-to`	The model the request was sent to.
`x-cave-route-tier`	Your pinned tier, present exactly when it narrowed the decision. Absent means no pin, or a pin the decision did not consult.
`x-cave-route-decision-latency-us`	What deciding cost, in microseconds.

When routing does not run#

Routing is a per-project capability, and a project can mark it opt-in. On an opt-in project, a request activates it by sending x-cave-optimize: route. Lossless and pass-through requests are never routed, because those modes promise byte-identical forwarding. And a named model is a named model: the router does not touch it.

What it will not do, and what its numbers mean#

The router does not claim savings. It records decisions; what a decision was worth is computed by the spend plane against traffic that actually happened, under the same vocabulary as everything else here: benchmark-derived rankings are measured on those benchmarks, your task-type rankings are measured on your graded traffic, and neither is verified by itself.

Where the public matrix has no evidence for a task and tier, the router abstains and uses your baseline rather than guessing. Abstaining is a routing decision and shows up in the receipts like any other.