Caveman Middleware: Framework Adapters for TS and Python

This quickstart targets middleware 1.0.1 and SDK 1.2.0. Read the release notes and limitations before upgrading.

Run a complete example#

Follow the AI SDK quickstart for a fresh environment.

npm install --save-exact @caveman-ai/sdk@1.2.0 @caveman-ai/middleware@1.0.1 ai@7.0.94 @ai-sdk/provider@4.0.11 zod@4.4.3
curl -fsSLo quickstart.ts https://docs.caveman.so/examples/middleware/quickstart.ts
DEMO_MODE=record node --experimental-strip-types quickstart.ts
DEMO_MODE=compress node --experimental-strip-types quickstart.ts
DEMO_MODE=off node --experimental-strip-types quickstart.ts

The default example makes no provider request. It runs a deterministic native model/tool loop against the real local runtime, checks compression and exact paginated recovery, and asserts that application history retains originals. The optional provider run is separately labeled and can incur charges.

Entry points: which ones compress#

Compression needs a recovery tool the adapter registered itself, so the model can always fetch an exact original. Entry points that cannot register one only record: in compress mode they pass content through unchanged, report recovery_unbound, and log once which entry point to use instead.

Certified adapters are gated by the conformance suite. Experimental ones get the same fail-open guard, logging and tests, but may change in a minor release. CavemanGoogleGenAI hooks the Google SDK's internal ApiClient; if that moves, the client falls back to the native one (adapter_error). CavemanDocumentCompressor (LangChain RAG) compresses only with a sourceExpansion reader. @caveman-ai/middleware/langchain loads langchain; if you only wrap a chat model or compress documents, import withCavemanModel, CavemanChatModel, CavemanDocumentCompressor and scopeFromConfig from @caveman-ai/middleware/langchain-model instead, which needs only @langchain/core. Every compressing entry point disables recovery if your tools already include one named caveman_retrieve; calls then report recovery_name_conflict.

Framework versions#

The package declares its frameworks as optional peers with no version range (*), so strict resolvers such as Yarn PnP let it import them. Ranged peers made a plain npm install fail with ERESOLVE (0.1.0-alpha.1), so the version check happens at run time instead, against the copy the adapter actually runs: for openai and @anthropic-ai/sdk, the version of the client you pass in; for every other framework, the copy this package resolves, whatever the working directory. When your application resolves a different copy of a framework than this package does, the gate still reads this package's copy and logs one warning naming both versions.

In a workspace (npm, pnpm, Yarn), @caveman-ai/middleware can be hoisted to the repository root next to an older copy of a framework that another package pulled in (ai@6, say) while your app uses its own ai@7. The adapter then runs ai@6 and passes through with unsupported_version. Make both resolve one copy: find the other one (npm ls ai, pnpm why ai), then dedupe it (npm dedupe, pnpm dedupe) or pin one version for the repository (overrides in npm, pnpm.overrides). Under pnpm with hoist=false, add the frameworks you use to public-hoist-pattern. @strands-agents/sdk itself declares peers on openai 6, @ai-sdk/provider 3 and @anthropic-ai/sdk 0.109, so npm needs --legacy-peer-deps to install it next to newer versions of those.

inspectFrameworkCompatibility(adapter) from @caveman-ai/middleware/compatibility reports the version of the copy this package resolves, supported range, tested releases and tier without importing a framework (it has no client, so for openai and anthropic it can differ from what the wrapped client reports). package.json lists the same data in supportedFrameworkVersions and testedFrameworkVersions.

• Out of range (including every prerelease, such as 7.1.0-canary.3): calls pass through unchanged, reporting unsupported_version, with one warning. After testing that version, set acceptFrameworkVersion: true in the adapter options.
• Unreadable (a deploy bundle with no node_modules): the adapter checks that the framework hooks it needs exist and runs, with one version_unverified warning. If a hook is missing it passes through with version_unavailable.
• Nothing throws when you wrap a client. With strict: true, the version decision surfaces from runtime.ready() / preflight().

Scope per request#

Every adapter takes scope as a value or a function. A function is called for each request, inside that request's async context, so one module-level client can serve many users, and runtime.deleteSession(scope) removes exactly one user's originals:

const client = withCavemanOpenAI(new OpenAI(), { runtime, fetch, scope: () => ({ namespace: 'app', session_id: currentUser().id }) });

LangChain passes the RunnableConfig (config => scopeFromConfig(config, 'app') reads configurable.thread_id), Mastra its RequestContext, Strands the agent. Any ID text works: values outside [A-Za-z0-9._:/-] (emails, user 42 / chat #7) become a stable h- hash. A missing scope (no thread_id) runs the call recovery-free with recovery_unbound. An unusable one reports invalid_scope. Neither throws, except invalid_scope in strict mode.

Budgets and logs#

A large history never skips the whole call. Each tool result over the runtime's segment_bytes is skipped on its own (payload_budget), and images or bytes enter the context manifest as hashes. manifestBytes (default 2 MiB) bounds how much history is hashed, and wireBytes (default 16 MiB, OpenAI and Anthropic) bounds the request body the adapter parses. OpenAI Responses turns pass through with provider_state_retained unless store: false, because OpenAI would store the compressed turn; allowStoredResponses: true opts in.

Each pass-through reason logs once per process through console.warn, as adapter=<id> reason=<code>, never content, scope values or credentials. Any exception inside adapter code becomes a pass-through with adapter_error; with strict: true the call raises MiddlewareError('adapter_error') instead.

Runtimes and module systems#

Node.js 22.12 or later, ESM import or CommonJS require(). TypeScript resolves the types with moduleResolution bundler, nodenext, node16 or node10. Edge runtimes (Cloudflare workerd, Vercel edge-light) are not supported: adapters rely on node:async_hooks call ownership, and importing one under those conditions throws an explanatory error. Run the adapter in a Node.js function instead.

This quickstart targets middleware 1.0.0 and SDK 1.2.0. Read the release notes and limitations before upgrading.

Run a complete example#

Follow the LangChain quickstart for a fresh environment.

python -m pip install 'caveman-sdk==1.2.0' 'caveman-middleware[langchain]==1.0.0' 'langchain==1.4.1' 'langchain-core==1.6.3' 'langgraph==1.2.11'
curl -fsSLo quickstart.py https://docs.caveman.so/examples/middleware/quickstart.py
DEMO_MODE=record python quickstart.py
DEMO_MODE=compress python quickstart.py
DEMO_MODE=off python quickstart.py

The default example makes no provider request. It runs a deterministic native model/tool loop against the real local runtime, checks compression and exact paginated recovery, and asserts that application history retains originals. The optional provider run is separately labeled and can incur charges.

Adapters, tiers, and what compresses#

Certified adapters (langchain, openai, anthropic, litellm) gate every release. Experimental adapters get the same fail-open guard, logging, and behavioral tests, but their APIs may change in a minor release. caveman_middleware.COMPATIBILITY exposes each family's tier and tested version range.

Compression needs a recovery tool the model can call, so only some entry points compress. The record-only ones never change input: in compress mode each call reports recovery_unbound, logged once.

If your app already has a tool named caveman_retrieve, that tool keeps its name, recovery stays off for that registration, and the adapter logs recovery_name_conflict once.

Providers#

• OpenAI: openai 2.x (on httpx) and 3.x (on httpx2). The adapter follows whichever HTTP library the installed SDK uses. A Responses API call that the provider would store (store defaults to true) is sent unchanged with reason provider_state_retained. Pass store=False, or allow_stored_responses=True to opt in.
• Anthropic: Anthropic, AnthropicBedrock, and AnthropicVertex, sync and async.
• LiteLLM: the async and proxy paths project any provider, because they edit LiteLLM's OpenAI-format input. The sync Router path edits the translated request and supports only the openai and anthropic providers; other providers pass through with reason unsupported_provider.
• LlamaIndex: the OpenAI and Anthropic LLMs, including subclasses such as AzureOpenAI. Bedrock Converse, Vertex, and other LLMs pass through with reason unsupported_provider.
• Pydantic AI: OpenAIChatModel (including Azure and OpenAI-compatible providers) and AnthropicModel (including Bedrock and Vertex clients). FallbackModel, Google/Gemini, Bedrock Converse, and other models pass through with reason unsupported_provider.
• Google: wrapping returns a clone, so your Client or Chat is never modified. Wrapping a wrapped object does not stack, and unwrap_google() returns the original. One Caveman transport serves every clone: each call carries the scope of the clone that made it.

Framework versions#

Each adapter checks the installed framework version against its tested range:

• Outside the range: the adapter skips (original input, reason unsupported_version) and logs a warning once. Pass accept_framework_version=True after testing a newer release yourself.
• Unreadable version (vendored or bundled builds): the adapter runs and logs version_unverified once.
• Too old or too new to import: importing the adapter raises an ImportError that names the installed version and the tested range, with code == "unsupported_version".

Wrapping never raises for a version problem, even in strict mode. To fail fast at startup, call caveman_middleware.preflight(runtime, "langchain"), which reports unavailable for an untested framework. caveman_middleware.ready(...) raises instead.

[openai] accepts openai 2.20 through 3.x. LiteLLM and CrewAI require openai<3, so [openai,litellm] or [openai,crewai] installs openai 2.x. [asgi] has no dependencies; it works with any ASGI 3 server or framework.

Failures, scopes, and logging#

• Fail-open: outside strict mode, any adapter or runtime failure sends your original request and reports a reason (adapter_error for a bug in adapter code). In strict mode those reasons raise MiddlewareError, except version problems.
• One warning per problem: every adapter logs one WARNING per adapter and reason on the caveman.middleware logger. The line never contains content, scope values, or credentials.
• Recovery errors: when the model calls caveman_retrieve with an unknown or expired handle, or the runtime is unavailable, the tool answers with an error JSON through the framework's own tool-error result and the run continues, even in strict mode. Cancellation still propagates.
• Scopes: thread and session IDs may be free-form. An email or user 42 / chat #7 is hashed into a valid scope token. A missing scope, such as scope_from_config without a thread_id, passes through with reason invalid_scope; strict mode raises.
• Runtimes: every adapter accepts either MiddlewareRuntime or AsyncMiddlewareRuntime and uses the matching view on sync and async paths.
• Long histories: history hashing stops at 2 MiB (manifest_bytes) and images or bytes are hashed. The ASGI adapter's body limit is max_body_bytes (default 2 MiB); a larger body passes through with reason payload_budget.