> ## Documentation Index
> Fetch the complete documentation index at: https://docs.caveman.so/llms.txt
> Use this file to discover all available pages before exploring further.

# Caveman Engine: Local Context Compression for AI Agents

> Learn how Caveman Engine detects, compresses, and recovers agent context. Covers the compression pipeline, 16 built-in compressors, recovery, token accounting, and CLI usage.

Caveman Engine is a local Go library and command-line runtime for reducing model context. It detects input shape, chooses a matching compressor, stores exact source when a lossy transform needs recovery, and emits output only when the result passes size and safety checks. Engine runs locally without a Caveman account.

## Library calls

Four operations form the stable core API:

| Operation | Purpose |
| - | - |
| `Compress` | Detect and compress input under a mode and policy |
| `Retrieve` | Recover exact original bytes from a handle |
| `Detect` | Classify input without changing it |
| `Stats` | Read local compression statistics |

`Simulate` is an auxiliary operation that evaluates a transform without committing normal runtime effects.

Callers should treat original-byte output as a valid result. It means no eligible transform passed all gates.

## Compression pipeline

```mermaid theme={"dark"}
flowchart LR
    A["Input bytes"] --> B["Detect shape"]
    B --> C["Select compressor"]
    C --> D["Transform"]
    D --> E{"Smaller and valid?"}
    E -- No --> A
    E -- Yes --> F{"Recovery required?"}
    F -- No --> G["Emit compact bytes"]
    F -- Yes --> H["Store exact original"]
    H --> I{"Store succeeded?"}
    I -- No --> A
    I -- Yes --> G
```

No compressor, parse error, larger result, or recovery-store failure returns original bytes. Unknown modes also return original bytes. `record` mode always passes input through.

## Automatic detection

Automatic detection recognizes:

* JSON
* test reports (a JUnit XML root element, or top-level pytest-json-report or Jest JSON keys, that parse with at least one test)
* terminal output
* diffs
* HTML
* tabular data
* source code
* logs
* search results
* configuration text
* general text

Some compressors are explicit-only because automatic selection would be ambiguous or unsafe. TOON, accessibility trees, tool-schema transforms, repetition transforms, and tool-schema annotation transforms are not selected by general detection.

## Compressor registry

Default registry contains 16 compressors:

1. JSON
2. log
3. code
4. diff
5. search result
6. text
7. HTML
8. tabular
9. configuration
10. tool schema
11. tool-schema annotations
12. TOON
13. accessibility tree
14. repetition
15. terminal output
16. test report (JUnit, pytest, Jest): keeps the counts and every failure

The code compressor keeps imports, signatures, and type declarations and elides function bodies. A cgo build parses Go, Python, TypeScript, JavaScript, Rust, Java, C, and C++ with tree-sitter; the pure-Go build parses Go with the standard library. The release `caveman-proxy` and `caveman-engine` are cgo builds on all six targets, cross-compiled with a pinned zig (`scripts/build-release-binaries.mjs`); releases up to bin-v2.0.2 were pure-Go, so their code compression covered Go only. The other four release binaries, the container image, and any `CGO_ENABLED=0` source build are still pure-Go. GDScript is elided by a line scanner in both builds: the language is indentation-scoped and has no grammar in either.

Each compressor declares a safety class and implements its own parse and output rules. Current compressors belong to lossy class S4, even when a particular input can round-trip structurally. Callers must not infer byte safety from a compressor name.

## Recovery requirement

A lossy result may be emitted only when exact original bytes are recoverable. Native callers supply a recovery store. Browser builds use an in-memory store; host runtimes normally use SQLite. If no recovery store exists, Engine keeps original input.

See [Context recovery](/caveman/context-recovery) for handles, limits, and object pointers.

## Token accounting

Engine uses an offline `o200k_base` counter where available. A character-based estimate is the fallback. Local counts therefore use an `inferred` basis unless a provider supplies its own usage. They are useful for comparison and capacity planning, not provider billing reconciliation.

## Input limits

CLI compression accepts input up to 64 MiB. Recovery storage has a separate configured capacity and a default payload ceiling of 512 MiB. Inputs exceeding limits are rejected or passed through according to caller contract; they are never silently truncated.

## Native and WebAssembly builds

Engine can run as:

* a Go library
* a native CLI used by the JavaScript launcher
* a WebAssembly module for browser-compatible consumers

Build native Engine from repository root:

```bash theme={"dark"}
go build ./engine/cmd/caveman-engine
```

Run Engine tests:

```bash theme={"dark"}
go test ./engine/...
```

WebAssembly uses in-memory recovery because browser runtimes do not expose the host SQLite store directly.

## Engine CLI

```text theme={"dark"}
caveman-engine compress < input
caveman-engine detect < input
caveman-engine retrieve <handle>
caveman-engine stats
caveman-engine registry
caveman-engine toon encode|decode
caveman-engine pixel render|simulate
caveman-engine evals run [--fixtures DIR]
```

Commands reading stdin enforce a 64 MiB input limit and fail with `cave_input_too_large` above it. Caller-supplied eval fixtures are confined to `DIR`; traversal and escaping symlinks fail closed.

Persistent CCR storage retains at most 512 MiB of payloads by default. Set `CAVEMAN_CCR_MAX_BYTES` to a positive byte count before startup to choose a different cap. At cap, new lossy transforms fail closed to original-byte pass-through; existing handles are never evicted and remain recoverable.

## Adding a compressor

A new compressor needs:

1. an unambiguous name and declared safety class
2. bounded parsing and deterministic output
3. original-byte fallback on invalid input
4. recovery integration for lossy output
5. fixtures covering valid, invalid, larger-output, and boundary cases
6. registration in the default registry only when automatic or explicit selection behavior is clear

Public performance claims require committed fixtures, method, token basis, and quality checks. See [Testing and benchmarks](https://github.com/JuliusBrussee/caveman/blob/main/docs/technical/testing-and-benchmarks.md).

Engine source is licensed under Apache-2.0, like the rest of the repository. See [`LICENSING.md`](https://github.com/JuliusBrussee/caveman/blob/main/LICENSING.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.