---
title: LiteLLM
summary: In front of LiteLLM, behind it, or as a callback.
canonical: https://docs.caveman.so/docs/proxy/litellm
layer: proxy
license: MIT
capability: integration-recipes
updated: 2026-09-16T21:32:40-07:00
basis: inferred
---

# LiteLLM

> In front of LiteLLM, behind it, or as a callback.
LiteLLM and the caveman proxy both speak the OpenAI wire protocol, so either can sit in front of the other.
Put LiteLLM in front when it already owns your keys, your fallbacks and your model aliases. Put caveman in
front when you want one entry point whose compression applies to every model LiteLLM can reach.

## LiteLLM in front

One argument per call, `api_base`.

```python
import os
import litellm

res = litellm.completion(
    model="openai/gpt-5.5",
    api_base="http://127.0.0.1:8787/w/my-service/openai/v1",
    api_key=os.environ["OPENAI_API_KEY"],
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
```

Printed by `caveman snippets litellm --app my-service`. `/w/my-service` is the attribution slug, so every row
the proxy records for this deployment carries that name.

For the LiteLLM proxy, the same change goes in `config.yaml` once per model:

```yaml
model_list:
  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_base: http://127.0.0.1:8787/w/my-service/openai/v1
      api_key: os.environ/OPENAI_API_KEY
```

LiteLLM keeps the provider key, the retries and the routing. Requests for that model leave LiteLLM, reach the
caveman proxy, get compressed if the mode allows it, and go upstream to OpenAI.

## caveman in front

A `compat` mount points a caveman route at a running LiteLLM proxy. Your application talks to caveman only.

```yaml
# ~/.caveman/caveman.yaml
mode: compress
compat:
  litellm:
    base_url: http://127.0.0.1:4000/v1
    api_key_env: LITELLM_MASTER_KEY
```

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8787/compat/litellm",
    api_key=os.environ["LITELLM_MASTER_KEY"],
)

res = client.chat.completions.create(
    model="gpt-5.5",
    messages=[{"role": "user", "content": "Why is the sky blue?"}],
)
```

```bash
CAVE_SSRF_ALLOWLIST=localhost:4000 caveman start
```

The proxy blocks loopback and private-range destinations at dial time, so a LiteLLM instance on the same
machine needs that host in `CAVE_SSRF_ALLOWLIST`. Leave it out and the mount answers 502 with nothing added
to `proxy.log`:

```json
{"error":{"type":"cave_gateway_error","code":"cave_upstream_unavailable","message":"Upstream service error.","request_id":"…"}}
```

Captured from caveman-proxy bin-v1.1.7 against a compat mount pointed at a loopback upstream, with
`request_id` replaced by `…`. [Where the proxy dials](/docs/proxy/security#where-the-proxy-dials) has the
grammar of an allowlist entry and what no entry can reach.

The mount is named, so the path carries the name: `/compat/litellm/chat/completions`. Everything under
[Modes](/docs/proxy/modes) applies to it, and compression runs before the bytes reach LiteLLM.

## The callback

A third placement would leave inference with LiteLLM and ask Caveman only which deployment alias to use: the
registry lists a LiteLLM callback package, `caveman-router`, as a public capability. Its source mirror has
not shipped, so the package is on neither PyPI nor the public repository today.

## Which one to pick

| You want | Placement |
|---|---|
| Compression, with LiteLLM keeping keys and fallbacks | LiteLLM in front |
| Compression, with one entry point for every model | caveman in front |