LiteLLM

LiteLLM and the caveman proxy both speak the OpenAI wire protocol, so either can sit in front of the other. Put LiteLLM in front when it already owns your keys, your fallbacks and your model aliases. Put caveman in front when you want one entry point whose compression applies to every model LiteLLM can reach.

LiteLLM in front#

One argument per call, api_base.

import os
import litellm

res = litellm.completion(
    model="openai/gpt-5.5",
    api_base="http://127.0.0.1:8787/w/my-service/openai/v1",
    api_key=os.environ["OPENAI_API_KEY"],
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)

Printed by caveman snippets litellm --app my-service. /w/my-service is the attribution slug, so every row the proxy records for this deployment carries that name.

For the LiteLLM proxy, the same change goes in config.yaml once per model:

model_list:
  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_base: http://127.0.0.1:8787/w/my-service/openai/v1
      api_key: os.environ/OPENAI_API_KEY

LiteLLM keeps the provider key, the retries and the routing. Requests for that model leave LiteLLM, reach the caveman proxy, get compressed if the mode allows it, and go upstream to OpenAI.

caveman in front#

A compat mount points a caveman route at a running LiteLLM proxy. Your application talks to caveman only.

# ~/.caveman/caveman.yaml
mode: compress
compat:
  litellm:
    base_url: http://127.0.0.1:4000/v1
    api_key_env: LITELLM_MASTER_KEY

import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8787/compat/litellm",
    api_key=os.environ["LITELLM_MASTER_KEY"],
)

res = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)

CAVE_SSRF_ALLOWLIST=localhost:4000 caveman start

The proxy blocks loopback and private-range destinations at dial time, so a LiteLLM instance on the same machine needs that host in CAVE_SSRF_ALLOWLIST. Leave it out and the mount answers 502 with nothing added to proxy.log:

{"error":{"type":"cave_gateway_error","code":"cave_upstream_unavailable","message":"Upstream service error.","request_id":"…"}}

Captured from caveman-proxy bin-v1.1.7 against a compat mount pointed at a loopback upstream, with request_id replaced by …. Where the proxy dials has the grammar of an allowlist entry and what no entry can reach.

The mount is named, so the path carries the name: /compat/litellm/chat/completions. Everything under Modes applies to it, and compression runs before the bytes reach LiteLLM.

The callback#

A third placement would leave inference with LiteLLM and ask Caveman only which deployment alias to use: the registry lists a LiteLLM callback package, caveman-router, as a public capability. Its source mirror has not shipped, so the package is on neither PyPI nor the public repository today.

Which one to pick#