---
title: Deploy
summary: Running the runtime beside your app, keeping recovery working across restarts, and measuring the result.
canonical: https://docs.caveman.so/docs/sdk/middleware/deployment
license: MIT
capability: sdk-ts
updated: 2026-09-16T21:32:40-07:00
basis: inferred
---

# Deploy

> Running the runtime beside your app, keeping recovery working across restarts, and measuring the result.
The middleware runtime is the Caveman proxy. It sits next to your application, answers `/caveman/v1/middleware/*`, and keeps the originals it stored on its own disk. Every optimize call is on the path of a provider call, so run it close: same host, same pod, or the same container network.

## Run it as a process

```bash
CAVEMAN_MODE=compress caveman start
```

`caveman start` binds 127.0.0.1:8787. `--port` and `--host` move it, `--config` points at a different `caveman.yaml`. `CAVEMAN_MODE=compress` is what puts it in compress mode; on its own it runs `record`, which measures and hands every byte back as it arrived.

## Run it as a container

The public repository ships a Dockerfile that builds the proxy into a distroless image.

```bash
git clone https://github.com/JuliusBrussee/caveman.git
cd caveman
docker build -t caveman-proxy .
docker run -e CAVEMAN_MODE=compress -p 8787:8787 -v caveman-data:/data caveman-proxy
```

- Listen: `CAVEMAN_LISTEN=0.0.0.0:8787`
- State: `CAVEMAN_HOME=/data`
- User: uid 65532, non-root
- Probe: `GET /health/ready`
- Shell: None. The image has no shell and no HEALTHCHECK, so probe from the orchestrator

A named volume inherits the image's ownership. For a bind mount, chown the host directory to 65532 first, so the proxy can create its store under `/data`.

## Point the application at it

Pass `endpoint` yourself. The runtime client resolves it from that argument alone, so any environment variable is yours to name and read.

```ts
const runtime = createMiddlewareRuntime({
  endpoint: process.env.CAVE_RUNTIME_URL ?? "http://127.0.0.1:8787",
  mode: "compress",
  deadlineMs: 500,
});
```

`CAVE_RUNTIME_URL` is the name the examples on this site use, read by the example itself.

## A runtime on another host

Two things change once the endpoint stops being loopback.

Give the runtime an inbound token. It refuses any non-loopback `CAVEMAN_LISTEN` without one, because that bind would otherwise expose every configured provider credential. The token is at least 16 bytes and carries no spaces or control characters, and it is read only from the environment.

```bash
CAVEMAN_AUTH_TOKEN="$(openssl rand -hex 24)" CAVEMAN_LISTEN=0.0.0.0:8787 CAVEMAN_MODE=compress caveman start
```

Give the client the same token, TLS, and explicit consent to send content off the machine.

```ts
const runtime = createMiddlewareRuntime({
  endpoint: "https://caveman.internal.example",
  token: process.env.CAVE_RUNTIME_TOKEN,
  allowRemoteContent: true,
  mode: "compress",
});
```

A non-loopback endpoint without `allowRemoteContent: true` and `https:` throws `remote_content_not_enabled` at construction. The token authenticates to the runtime. Mint it for that job alone, separate from any model provider credential.

The client refuses redirects, so give it the runtime's origin directly. A redirect produces `redirect_refused`.

## Serverless

A serverless function cannot hold a sidecar: the runtime is a process with a store on disk, and the
invocation ends before either is worth building. Point the function at a runtime on a host you keep running,
over `https:` with `allowRemoteContent: true` and `CAVEMAN_AUTH_TOKEN` as above, or run the function without
compression. With no reachable endpoint three calls report `runtime_unavailable`, the third opens the
circuit for 30 seconds so the calls after it report `circuit_open` without asking, and the bytes go out as
the framework built them throughout.

## Keep the scope stable

`scope` is four strings: `namespace`, `session_id`, `branch_id`, `cache_epoch`. Together they key the stored originals. A handle the model was given in one turn only resolves under the same four, so a scope that changes between turns breaks recovery even though nothing else looks wrong.

| Part | What it should be |
|---|---|
| `namespace` | Your tenant or application. Never shared across end users |
| `session_id` | The conversation id, stable for the whole conversation |
| `branch_id` | The branch of that conversation, `main` when there is one |
| `cache_epoch` | Bumped on purpose to start fresh choices, never per request |

Derive all four from state you already persist, so a restart hands the model the same `session_id` and the handles it is holding still resolve.

## How long originals live

A `cmw_` handle is good for 86,400 seconds, and every optimize or retrieve that touches its scope pushes the expiry out by that much again. The idle clock is what expires it, so a conversation that keeps referring to the same tool result keeps it alive; a day of silence lets it lapse. `capabilities.retention_seconds` reports the number the runtime is actually using.

A recovery call after expiry returns `expired`. The compressed copy stays in the model's history, so the model keeps the shortened view and loses the way back to the full text.

That clock governs the handle, not the bytes. The original itself sits in the proxy's recovery store under a content-addressed `ccr_` handle the application never sees, and nothing evicts it. [Recoverable compression](/docs/proxy/recoverable#two-stores) has that store and its budget.

## What to measure

Two numbers, and both have to move together.

- **Task success on your own evaluation.** Compression changes what the model reads. A shorter context that costs you a wrong answer is not a saving.
- **Total provider usage for the whole task, recovery calls included.** A recovery call is a provider round trip carrying the original back. Counting only the first call makes the result look better than it is.

`onReport` gives you the local decision per call, and `runtime.observe(receipt)` sends client-observed usage to the runtime as metadata. Both are `inferred`. [Numbers and limits](/docs/counting) defines the word.

## Turn it off

Restart the runtime without `CAVEMAN_MODE=compress` and it goes back to `record`, measuring every call and handing the bytes back as they arrived. `mode: "off"` on the client stops even that, from the runtime options and with no change at the call site. [Middleware](/docs/sdk/middleware) has the call-site undo.