> ## Documentation Index
> Fetch the complete documentation index at: https://docs.caveman.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Caveman Middleware: Framework Adapters for TS and Python

> Add Caveman compression to your AI framework with TypeScript and Python middleware adapters: entry points, scope, budgets, version gates, and recovery.

Caveman middleware provides native framework adapters for the Caveman compression runtime. Your framework keeps its inference client, tools, retries, streams, and original conversation. Caveman projects eligible tool-result text into a copied outbound request, while inference stays with your provider.

Install the client packages:

<CodeGroup>
  ```bash TypeScript theme={"dark"}
  npm install @caveman-ai/middleware @caveman-ai/sdk
  ```

  ```bash Python theme={"dark"}
  pip install 'caveman-middleware[langchain]' caveman-sdk
  ```
</CodeGroup>

<Note>
  Start the local runtime separately; the client packages do not include it. See the hosted [runtime installation guide](https://docs.caveman.so/docs/sdk/middleware/deployment#run-it-as-a-process).
</Note>

<Tabs>
  <Tab title="TypeScript">
    This quickstart targets middleware `1.0.1` and SDK `1.2.0`. Read the [release notes and limitations](https://github.com/JuliusBrussee/caveman/blob/main/packages/middleware/typescript/CHANGELOG.md) before upgrading.

    ### Run a complete example

    Follow the [AI SDK quickstart](https://docs.caveman.so/docs/sdk/middleware/vercel-ai-sdk) for a fresh environment.

    ```bash theme={"dark"}
    npm install --save-exact @caveman-ai/sdk@1.2.0 @caveman-ai/middleware@1.0.1 ai@7.0.94 @ai-sdk/provider@4.0.11 zod@4.4.3
    curl -fsSLo quickstart.ts https://docs.caveman.so/examples/middleware/quickstart.ts
    DEMO_MODE=record node --experimental-strip-types quickstart.ts
    DEMO_MODE=compress node --experimental-strip-types quickstart.ts
    DEMO_MODE=off node --experimental-strip-types quickstart.ts
    ```

    The default example makes no provider request. It runs a deterministic native model/tool loop against the real local runtime, checks compression and exact paginated recovery, and asserts that application history retains originals. The optional provider run is separately labeled and can incur charges.

    ### Entry points: which ones compress

    Compression needs a recovery tool the adapter registered itself, so the model can always fetch an exact original. Entry points that cannot register one only record: in `compress` mode they pass content through unchanged, report `recovery_unbound`, and log once which entry point to use instead.

    | Adapter | Tier | Compresses | Records only (`recovery_unbound` in compress mode) |
    | - | - | - | - |
    | `ai-sdk` | certified | `withCaveman` | `createCavemanMiddleware` (`wrapLanguageModel`) |
    | `openai` | certified | `withCavemanOpenAITools`, `chat.completions.runTools` on `withCavemanOpenAI` | plain `create` calls on `withCavemanOpenAI` |
    | `anthropic` | certified | `beta.messages.toolRunner` on `withCavemanAnthropic` | `messages.create` / `messages.stream` |
    | `langchain` | certified | `withCavemanAgent`, `createCavemanLangChain` | `withCavemanModel`, `CavemanChatModel` |
    | `google` | experimental | `CavemanGoogleGenAI` with callable tools (automatic function calling) | calls without callable tools |
    | `strands` | experimental | `withCavemanStrands` | `withCavemanStrandsModel` |
    | `mastra` | experimental | `withCavemanMastra` | `createCavemanMastraProcessor` |
    | `mcp` | experimental | `CavemanMCPHost` with `register()`ed tools | |

    Certified adapters are gated by the conformance suite. Experimental ones get the same fail-open guard, logging and tests, but may change in a minor release. `CavemanGoogleGenAI` hooks the Google SDK's internal `ApiClient`; if that moves, the client falls back to the native one (`adapter_error`). `CavemanDocumentCompressor` (LangChain RAG) compresses only with a `sourceExpansion` reader. `@caveman-ai/middleware/langchain` loads `langchain`; if you only wrap a chat model or compress documents, import `withCavemanModel`, `CavemanChatModel`, `CavemanDocumentCompressor` and `scopeFromConfig` from `@caveman-ai/middleware/langchain-model` instead, which needs only `@langchain/core`. Every compressing entry point disables recovery if your tools already include one named `caveman_retrieve`; calls then report `recovery_name_conflict`.

    ### Framework versions

    The package declares its frameworks as optional peers with no version range (`*`), so strict resolvers such as Yarn PnP let it import them. Ranged peers made a plain `npm install` fail with `ERESOLVE` (0.1.0-alpha.1), so the version check happens at run time instead, against the copy the adapter actually runs: for `openai` and `@anthropic-ai/sdk`, the version of the client you pass in; for every other framework, the copy this package resolves, whatever the working directory. When your application resolves a different copy of a framework than this package does, the gate still reads this package's copy and logs one warning naming both versions.

    In a workspace (npm, pnpm, Yarn), `@caveman-ai/middleware` can be hoisted to the repository root next to an older copy of a framework that another package pulled in (`ai@6`, say) while your app uses its own `ai@7`. The adapter then runs `ai@6` and passes through with `unsupported_version`. Make both resolve one copy: find the other one (`npm ls ai`, `pnpm why ai`), then dedupe it (`npm dedupe`, `pnpm dedupe`) or pin one version for the repository (`overrides` in npm, `pnpm.overrides`). Under pnpm with `hoist=false`, add the frameworks you use to `public-hoist-pattern`. `@strands-agents/sdk` itself declares peers on `openai` 6, `@ai-sdk/provider` 3 and `@anthropic-ai/sdk` 0.109, so npm needs `--legacy-peer-deps` to install it next to newer versions of those.

    `inspectFrameworkCompatibility(adapter)` from `@caveman-ai/middleware/compatibility` reports the version of the copy this package resolves, supported range, tested releases and tier without importing a framework (it has no client, so for `openai` and `anthropic` it can differ from what the wrapped client reports). `package.json` lists the same data in `supportedFrameworkVersions` and `testedFrameworkVersions`.

    * **Out of range** (including every prerelease, such as `7.1.0-canary.3`): calls pass through unchanged, reporting `unsupported_version`, with one warning. After testing that version, set `acceptFrameworkVersion: true` in the adapter options.
    * **Unreadable** (a deploy bundle with no `node_modules`): the adapter checks that the framework hooks it needs exist and runs, with one `version_unverified` warning. If a hook is missing it passes through with `version_unavailable`.
    * Nothing throws when you wrap a client. With `strict: true`, the version decision surfaces from `runtime.ready()` / `preflight()`.

    ### Scope per request

    Every adapter takes `scope` as a value or a function. A function is called for each request, inside that request's async context, so one module-level client can serve many users, and `runtime.deleteSession(scope)` removes exactly one user's originals:

    ```ts theme={"dark"}
    const client = withCavemanOpenAI(new OpenAI(), { runtime, fetch, scope: () => ({ namespace: 'app', session_id: currentUser().id }) });
    ```

    LangChain passes the `RunnableConfig` (`config => scopeFromConfig(config, 'app')` reads `configurable.thread_id`), Mastra its `RequestContext`, Strands the agent. Any ID text works: values outside `[A-Za-z0-9._:/-]` (emails, `user 42 / chat #7`) become a stable `h-` hash. A missing scope (no `thread_id`) runs the call recovery-free with `recovery_unbound`. An unusable one reports `invalid_scope`. Neither throws, except `invalid_scope` in strict mode.

    ### Budgets and logs

    A large history never skips the whole call. Each tool result over the runtime's `segment_bytes` is skipped on its own (`payload_budget`), and images or bytes enter the context manifest as hashes. `manifestBytes` (default 2 MiB) bounds how much history is hashed, and `wireBytes` (default 16 MiB, OpenAI and Anthropic) bounds the request body the adapter parses. OpenAI Responses turns pass through with `provider_state_retained` unless `store: false`, because OpenAI would store the compressed turn; `allowStoredResponses: true` opts in.

    Each pass-through reason logs once per process through `console.warn`, as `adapter=<id> reason=<code>`, never content, scope values or credentials. Any exception inside adapter code becomes a pass-through with `adapter_error`; with `strict: true` the call raises `MiddlewareError('adapter_error')` instead.

    ### Runtimes and module systems

    Node.js 22.12 or later, ESM `import` or CommonJS `require()`. TypeScript resolves the types with `moduleResolution` `bundler`, `nodenext`, `node16` or `node10`. Edge runtimes (Cloudflare `workerd`, Vercel `edge-light`) are not supported: adapters rely on `node:async_hooks` call ownership, and importing one under those conditions throws an explanatory error. Run the adapter in a Node.js function instead.
  </Tab>

  <Tab title="Python">
    This quickstart targets middleware `1.0.0` and SDK `1.2.0`. Read the [release notes and limitations](https://github.com/JuliusBrussee/caveman/blob/main/packages/middleware/python/CHANGELOG.md) before upgrading.

    ### Run a complete example

    Follow the [LangChain quickstart](https://docs.caveman.so/docs/sdk/middleware/python) for a fresh environment.

    ```bash theme={"dark"}
    python -m pip install 'caveman-sdk==1.2.0' 'caveman-middleware[langchain]==1.0.0' 'langchain==1.4.1' 'langchain-core==1.6.3' 'langgraph==1.2.11'
    curl -fsSLo quickstart.py https://docs.caveman.so/examples/middleware/quickstart.py
    DEMO_MODE=record python quickstart.py
    DEMO_MODE=compress python quickstart.py
    DEMO_MODE=off python quickstart.py
    ```

    The default example makes no provider request. It runs a deterministic native model/tool loop against the real local runtime, checks compression and exact paginated recovery, and asserts that application history retains originals. The optional provider run is separately labeled and can incur charges.

    ### Adapters, tiers, and what compresses

    Certified adapters (`langchain`, `openai`, `anthropic`, `litellm`) gate every release. Experimental adapters get the same fail-open guard, logging, and behavioral tests, but their APIs may change in a minor release. `caveman_middleware.COMPATIBILITY` exposes each family's tier and tested version range.

    Compression needs a recovery tool the model can call, so only some entry points compress. The record-only ones never change input: in `compress` mode each call reports `recovery_unbound`, logged once.

    | Extra | Tier | Compresses | Record-only |
    | - | - | - | - |
    | `langchain` | certified | `with_caveman_agent`, or `CavemanMiddleware` with its `recovery_tool` in the agent's tools | `with_caveman_model`; `CavemanDocumentCompressor` without `source_expansion` |
    | `openai` | certified | `with_caveman_openai_tools` | `with_caveman_openai` |
    | `anthropic` | certified | `beta.messages.tool_runner` on a `with_caveman_anthropic` client | `messages.create` on that client |
    | `litellm` | certified | `CavemanLiteLLM` with `operator_recovery` | calls without `operator_recovery` |
    | `google` | experimental | `with_caveman_google` / `with_caveman_google_chat` with callable AFC tools, over the Caveman transports | calls without callable tools |
    | `strands` | experimental | `with_caveman_agent` | `with_caveman_model` |
    | `agno` | experimental | `with_caveman_agent` | `with_caveman_model` |
    | `crewai` | experimental | `with_caveman_agent` for an agent with tools | `with_caveman_llm` alone |
    | `pydantic-ai` | experimental | `CavemanCapability` on the `Agent` | `with_caveman_model` |
    | `autogen` | experimental | `with_caveman_agent`, or `CavemanWorkbench` with its client | `with_caveman_model` |
    | `llama-index` | experimental | `CavemanFunctionAgent`, `with_caveman_tools` | `with_caveman_model`; `CavemanNodePostprocessor` without `source_expansion` |
    | `mcp` | experimental | `CavemanMCPHost.register` plus `project_result` | none |
    | `asgi` | experimental | `CavemanASGIMiddleware` with an `ASGIContext.recovery` binding | the same middleware without one |

    If your app already has a tool named `caveman_retrieve`, that tool keeps its name, recovery stays off for that registration, and the adapter logs `recovery_name_conflict` once.

    ### Providers

    * **OpenAI:** openai 2.x (on `httpx`) and 3.x (on `httpx2`). The adapter follows whichever HTTP library the installed SDK uses. A Responses API call that the provider would store (`store` defaults to true) is sent unchanged with reason `provider_state_retained`. Pass `store=False`, or `allow_stored_responses=True` to opt in.
    * **Anthropic:** `Anthropic`, `AnthropicBedrock`, and `AnthropicVertex`, sync and async.
    * **LiteLLM:** the async and proxy paths project any provider, because they edit LiteLLM's OpenAI-format input. The sync `Router` path edits the translated request and supports only the `openai` and `anthropic` providers; other providers pass through with reason `unsupported_provider`.
    * **LlamaIndex:** the `OpenAI` and `Anthropic` LLMs, including subclasses such as `AzureOpenAI`. Bedrock Converse, Vertex, and other LLMs pass through with reason `unsupported_provider`.
    * **Pydantic AI:** `OpenAIChatModel` (including Azure and OpenAI-compatible providers) and `AnthropicModel` (including Bedrock and Vertex clients). `FallbackModel`, Google/Gemini, Bedrock Converse, and other models pass through with reason `unsupported_provider`.
    * **Google:** wrapping returns a clone, so your `Client` or `Chat` is never modified. Wrapping a wrapped object does not stack, and `unwrap_google()` returns the original. One Caveman transport serves every clone: each call carries the scope of the clone that made it.

    ### Framework versions

    Each adapter checks the installed framework version against its tested range:

    * **Outside the range:** the adapter skips (original input, reason `unsupported_version`) and logs a warning once. Pass `accept_framework_version=True` after testing a newer release yourself.
    * **Unreadable version** (vendored or bundled builds): the adapter runs and logs `version_unverified` once.
    * **Too old or too new to import:** importing the adapter raises an `ImportError` that names the installed version and the tested range, with `code == "unsupported_version"`.

    Wrapping never raises for a version problem, even in strict mode. To fail fast at startup, call `caveman_middleware.preflight(runtime, "langchain")`, which reports `unavailable` for an untested framework. `caveman_middleware.ready(...)` raises instead.

    | Extra | Accepted (gate and extra) | Oldest tested | Newest tested |
    | - | - | - | - |
    | `langchain` | langchain 1.1 to 2, langchain-core 1.1 to 2, langgraph 1.0.2 to 2 | 1.1.0 / 1.1.0 / 1.0.2 | 1.4.2 / 1.6.5 / 1.2.12 |
    | `openai` | openai 2.20 to 4 | 2.20.0 | 2.54.0, 3.19.2 |
    | `anthropic` | anthropic 1.0 to 2 | 1.0.0 | 1.8.0 |
    | `litellm` | litellm 1.95 to 2 | 1.95.0 | 1.102.1 |
    | `google` | google-genai 2.18 to 3 | 2.18.0 | 2.25.0 |
    | `strands` | strands-agents 1.43 to 2 | 1.43.0 | 1.57.0 |
    | `agno` | agno 3.0 to 4 | 3.0.0 | 3.0.11 |
    | `crewai` | crewai 1.15.3 to 2 | 1.15.3 | 1.15.22 |
    | `pydantic-ai` | pydantic-ai-slim 2.36 to 3 | 2.36.0 | 2.49.0 |
    | `autogen` | autogen-agentchat, -core, -ext 0.7 to 0.8 | 0.7.0 | 0.7.5 |
    | `llama-index` | llama-index-core 0.14.5 to 0.15 | 0.14.5 | 0.14.25 |
    | `mcp` | mcp 2.0 to 3 | 2.0.0 | 2.2.0 |
    | `asgi` | none (ASGI 3) | n/a | n/a |

    `[openai]` accepts openai 2.20 through 3.x. LiteLLM and CrewAI require `openai<3`, so `[openai,litellm]` or `[openai,crewai]` installs openai 2.x. `[asgi]` has no dependencies; it works with any ASGI 3 server or framework.

    ### Failures, scopes, and logging

    * **Fail-open:** outside strict mode, any adapter or runtime failure sends your original request and reports a reason (`adapter_error` for a bug in adapter code). In strict mode those reasons raise `MiddlewareError`, except version problems.
    * **One warning per problem:** every adapter logs one `WARNING` per adapter and reason on the `caveman.middleware` logger. The line never contains content, scope values, or credentials.
    * **Recovery errors:** when the model calls `caveman_retrieve` with an unknown or expired handle, or the runtime is unavailable, the tool answers with an error JSON through the framework's own tool-error result and the run continues, even in strict mode. Cancellation still propagates.
    * **Scopes:** thread and session IDs may be free-form. An email or `user 42 / chat #7` is hashed into a valid scope token. A missing scope, such as `scope_from_config` without a `thread_id`, passes through with reason `invalid_scope`; strict mode raises.
    * **Runtimes:** every adapter accepts either `MiddlewareRuntime` or `AsyncMiddlewareRuntime` and uses the matching view on sync and async paths.
    * **Long histories:** history hashing stops at 2 MiB (`manifest_bytes`) and images or bytes are hashed. The ASGI adapter's body limit is `max_body_bytes` (default 2 MiB); a larger body passes through with reason `payload_budget`.
  </Tab>
</Tabs>

## Contracts to keep

Keep original stored history. Register the actual recovery executor through the native helper; a tool schema alone does not attest recovery. Handles are scope-bound and expire according to runtime retention. A recovery the runtime refuses (unknown or expired handle, runtime down) answers the model with an error JSON object and warns once, so the tool loop keeps running; caller cancellation still propagates. Recoverability does not guarantee model quality.

Client modes are `off`, `record`, and `compress`. The client defaults to compression; the standalone runtime defaults to recording. Set both deliberately.

<Warning>
  Final decision reports contain status, reason, transform IDs, replacement/reuse counts, and call IDs. They contain no token counters. Local segment estimates are inferred; provider usage and billed savings are separate evidence. See [Accounting and evidence](/caveman/accounting-and-evidence).
</Warning>

Close the runtime client and native framework/provider resources at shutdown. Closing the client does not stop the runtime process. See [Deploy the proxy for a team](/caveman/deploy) before sharing a runtime across workers or tenants.

## Protocol and license

The full [middleware protocol](https://github.com/JuliusBrussee/caveman/blob/main/docs/technical/middleware-protocol.md) covers routes, authentication, scope normalization, feature negotiation, error semantics, reason codes, budgets, and transport rules.

The client, adapters, and Engine runtime are all Apache-2.0. Read [LICENSING.md](https://github.com/JuliusBrussee/caveman/blob/main/LICENSING.md). File sanitized reproducible issues in [Caveman issues](https://github.com/JuliusBrussee/caveman/issues).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.