Skip to main content
Caveman Engine is a local Go library and command-line runtime for reducing model context. It detects input shape, chooses a matching compressor, stores exact source when a lossy transform needs recovery, and emits output only when the result passes size and safety checks. Engine runs locally without a Caveman account.

​
Library calls

Four operations form the stable core API: Simulate is an auxiliary operation that evaluates a transform without committing normal runtime effects. Callers should treat original-byte output as a valid result. It means no eligible transform passed all gates.

​
Compression pipeline

No compressor, parse error, larger result, or recovery-store failure returns original bytes. Unknown modes also return original bytes. record mode always passes input through.

​
Automatic detection

Automatic detection recognizes:
  • JSON
  • test reports (a JUnit XML root element, or top-level pytest-json-report or Jest JSON keys, that parse with at least one test)
  • terminal output
  • diffs
  • HTML
  • tabular data
  • source code
  • logs
  • search results
  • configuration text
  • general text
Some compressors are explicit-only because automatic selection would be ambiguous or unsafe. TOON, accessibility trees, tool-schema transforms, repetition transforms, and tool-schema annotation transforms are not selected by general detection.

​
Compressor registry

Default registry contains 16 compressors:
  1. JSON
  2. log
  3. code
  4. diff
  5. search result
  6. text
  7. HTML
  8. tabular
  9. configuration
  10. tool schema
  11. tool-schema annotations
  12. TOON
  13. accessibility tree
  14. repetition
  15. terminal output
  16. test report (JUnit, pytest, Jest): keeps the counts and every failure
The code compressor keeps imports, signatures, and type declarations and elides function bodies. A cgo build parses Go, Python, TypeScript, JavaScript, Rust, Java, C, and C++ with tree-sitter; the pure-Go build parses Go with the standard library. The release caveman-proxy and caveman-engine are cgo builds on all six targets, cross-compiled with a pinned zig (scripts/build-release-binaries.mjs); releases up to bin-v2.0.2 were pure-Go, so their code compression covered Go only. The other four release binaries, the container image, and any CGO_ENABLED=0 source build are still pure-Go. GDScript is elided by a line scanner in both builds: the language is indentation-scoped and has no grammar in either. Each compressor declares a safety class and implements its own parse and output rules. Current compressors belong to lossy class S4, even when a particular input can round-trip structurally. Callers must not infer byte safety from a compressor name.

​
Recovery requirement

A lossy result may be emitted only when exact original bytes are recoverable. Native callers supply a recovery store. Browser builds use an in-memory store; host runtimes normally use SQLite. If no recovery store exists, Engine keeps original input. See Context recovery for handles, limits, and object pointers.

​
Token accounting

Engine uses an offline o200k_base counter where available. A character-based estimate is the fallback. Local counts therefore use an inferred basis unless a provider supplies its own usage. They are useful for comparison and capacity planning, not provider billing reconciliation.

​
Input limits

CLI compression accepts input up to 64 MiB. Recovery storage has a separate configured capacity and a default payload ceiling of 512 MiB. Inputs exceeding limits are rejected or passed through according to caller contract; they are never silently truncated.

​
Native and WebAssembly builds

Engine can run as:
  • a Go library
  • a native CLI used by the JavaScript launcher
  • a WebAssembly module for browser-compatible consumers
Build native Engine from repository root:
Run Engine tests:
WebAssembly uses in-memory recovery because browser runtimes do not expose the host SQLite store directly.

​
Engine CLI

Commands reading stdin enforce a 64 MiB input limit and fail with cave_input_too_large above it. Caller-supplied eval fixtures are confined to DIR; traversal and escaping symlinks fail closed. Persistent CCR storage retains at most 512 MiB of payloads by default. Set CAVEMAN_CCR_MAX_BYTES to a positive byte count before startup to choose a different cap. At cap, new lossy transforms fail closed to original-byte pass-through; existing handles are never evicted and remain recoverable.

​
Adding a compressor

A new compressor needs:
  1. an unambiguous name and declared safety class
  2. bounded parsing and deterministic output
  3. original-byte fallback on invalid input
  4. recovery integration for lossy output
  5. fixtures covering valid, invalid, larger-output, and boundary cases
  6. registration in the default registry only when automatic or explicit selection behavior is clear
Public performance claims require committed fixtures, method, token basis, and quality checks. See Testing and benchmarks. Engine source is licensed under Apache-2.0, like the rest of the repository. See LICENSING.md.