Caveman benchmarks and honest numbers

Caveman publishes measured results with full provenance and explicit limits. This page covers the CaveBench Wrap benchmark, the response skill’s measured output reduction, per-file compression results, community A/B comparisons, and how to run your own benchmarks.

CaveBench Wrap benchmark

On six deterministic, agent-shaped tool-output workloads, Caveman-wrapped Claude Code used 33.2% fewer provider-reported input tokens than direct Claude Code: 591,673 vs 885,793 tokens across 18 paired runs. Caveman passed all 18/18 exact-answer checks. The case-clustered 95% interval was 14.6% to 48.5%.

Claim basis: benchmark_counterfactual. This is controlled benchmark evidence, not production traffic, customer spend, a provider invoice, or Caveman verified_savings.

Arm	Exact quality	Provider input on held pairs	Reduction vs direct	Case-clustered 95% interval
Direct Claude Code	18/18	885,793	baseline	n/a
Caveman wrap + skill	18/18	591,673	33.2%	14.6% to 48.5%
Headroom wrap	15/18	703,202 vs matched direct	6.7%	-0.7% to 17.9%

Caveman won 15/18 pairs. Headroom’s three YAML runs failed the exact-answer gate and remain visible rather than counting toward savings at held quality.

Per-case results

Each case ran three times per arm.

Case	Shape	Direct input	Caveman input	Reduction	Caveman quality
`sre-log-needle`	log	148,807	74,068	50.2%	3/3
`deployment-json-drift`	JSON	147,975	108,939	26.4%	3/3
`fraud-csv-outlier`	CSV	165,823	74,484	55.1%	3/3
`test-output-failure`	test output	150,377	108,514	27.8%	3/3
`config-yaml-drift`	YAML	132,124	71,027	46.2%	3/3
`dashboard-html-alert`	HTML	140,687	154,641	-9.9%	3/3

Unsupported and no-op inputs stay in the aggregate. HTML regressed because no compression transform applied while full Caveman skill overhead remained counted.

Method

Six immutable MCP fixtures, each 60–95 KB: logs; deployment JSON; fraud CSV; test output; configuration YAML; dashboard HTML.
Three rotated repetitions for direct Claude Code, Caveman, and Headroom: 54 total agent runs and 18 direct/Caveman pairs.
Claude Code 2.1.223, model claude-sonnet-5.
Headroom 0.33.0, pinned at upstream commit e0ce4b1d.
Every arm called the same fixture exactly once and returned a structured answer checked by an exact semantic JSON oracle.
Every arm used Claude Code’s modelUsage counters. Primary input metric: input_tokens + cache_read_input_tokens + cache_creation_input_tokens.
Cache buckets were summed without price weighting. Wrapper-native tokenizer estimates did not drive the comparison.
Recovery was available only when required answer data was absent from visible compressed content. Recovery calls and follow-up provider input remained counted.
Full Caveman skill prompt overhead remained counted from the first request.
Aggregate reduction compares summed Caveman input with summed paired direct input. The interval uses a deterministic 10,000-resample percentile bootstrap clustered by case.

Verification and provenance

Generated: 2026-08-06T14:31:43Z
Publication gate: passed
Same provider-usage source: 54/54 runs
Fixture called exactly once: 54/54 runs
Caveman skill installed, loaded, and applied: 18/18 runs
Positive proxy compression observed: 15/18 Caveman runs; no-op runs remained included
Permission denials: 0
Corpus SHA-256: 9a400a6dc38591dc3ce59bc2e3fa6fc59d99e211dc9185b430979de78991760a
Caveman skill SHA-256: 5e30bb56afbd0b01bd736f2da84180e76f18db4a64de8e124525d5c8dc2e8605
Harness source SHA-256: e6322a3a55cfdfa6c7942022a1be9adb49fa30c380352c819c99bbd4bff30cc4
Fixture MCP binary SHA-256: 5c71768780582708c00fa1d6862a6a5b93fc33fc23655ff430b0edb8fee1e790
Claude binary SHA-256: 4163c57c719e27680336f323ebdcd2ba8aa48a683fdfa427240ec4b506a21e45
Harness Git commit: 630e157246b68b63559fb8baab29b87042db996b
Dirty worktree at execution: false

Reproduction availability

This repository contains the published report and provenance hashes, but not raw harness or run artifacts for this result. It cannot be independently reproduced from this checkout. Treat the result as a pinned report, not a reproducible public benchmark, until harness and raw artifacts are published here. Publication requires at least six cases and three repetitions; exact direct and Caveman quality on every run; one fixture call per run; the same provider usage source; active Caveman skill; at least one proven compression; positive aggregate reduction; and a 95% interval entirely above zero. Negative and no-op cases may not be removed.

Boundaries

Workloads are deterministic large tool outputs, not open-ended coding tasks or customer traffic.
Result applies to this pinned suite and runtime. It is not a universal savings promise.
Local host isolation reset a dedicated Claude configuration before every arm. This was not a sealed container, and no exact filesystem or egress isolation is claimed.
Cost, latency, and output-token deltas are secondary because agent trajectories can differ. Primary claim is paired provider input tokens at held quality.

Response skill measured numbers

The Caveman response skill makes the model write shorter output. The skill itself does not compress input, context, files, or model thinking tokens. The Caveman local Engine and proxy are separate components that can compress recoverable input.

What	Number	How measured	Source
Output reduction vs a plain `Answer concisely.` control	caveman 3%, ultracave 35%, megacave 9% at the median (n=10, single run, output length only, tiktoken o200k approximation; measured on the 3.1.0 skill text, not re-run since the 3.2.0 rule changes)	`evals/llm_run.py` on claude-opus-5-5 with host settings isolated; `evals/measure.py`	`evals/snapshots/results.json`
Output reduction vs default verbose replies (no benchmark harness run)	Not published	`benchmarks/` harness exists, but the repository has no committed reviewed raw result	`benchmarks/`
Input reduction from the skill	0%	It’s an output-style instruction	Not applicable
Input cost the skill adds	Not measured yet	`evals/llm_run.py` now stores the usage Claude Code reports for every call, and `evals/measure.py` reports the median input tokens the skill adds against the terse control. The committed snapshot predates usage capture.	`evals/README.md`
`/caveman-compress` on memory files	33.2% total input reduction across five listed fixtures (22.8–49.1% per file, tiktoken o200k)	`skills/caveman-compress/scripts/benchmark.py` over `tests/caveman-compress/`, plus structural checks; no general quality-equivalence claim	caveman-compress fixtures

Token-count runs measure output length only. They do not prove semantic or technical equivalence. Publish a reduction only with committed raw pairs and separate quality review. A fidelity eval now exists (evals/score_fidelity.py: fixed cases, regex checks for kept numbers, negations, language and safety wording, scored for every arm), but no reviewed fidelity snapshot is committed, so no quality claim is published.

Per-file compression

Measured 2026-10-03, outside any agent run. Each of the six corpus files was regenerated byte for byte (corpus SHA-256 matches the one above) and passed once through engine.Simulate in compress mode, with the case question as the query. That is the path the proxy’s query-aware CompressSegmentQuery takes. Tokens are counted with the engine’s o200k_base counter. No model call, no recovery.

Case	Shape	File tokens	Today’s engine (caveman `aeb45e2f`)	Engine at the run above (Caveman-Cloud `630e1572`, `public/`)
`fraud-csv-outlier`	CSV	28,041	314 (98.9% smaller)	170 (99.4% smaller)
`sre-log-needle`	log	22,810	348 (98.5% smaller)	224 (99.0% smaller)
`config-yaml-drift`	YAML	20,447	178 (99.1% smaller)	178 (99.1% smaller)
`test-output-failure`	test output	18,806	203 (98.9% smaller)	203 (98.9% smaller)
`deployment-json-drift`	JSON	18,837	281 (98.5% smaller)	8,106 (57.0% smaller)
`dashboard-html-alert`	HTML	21,670	21,670 (no compressor)	21,670 (no compressor)
All six		130,611	22,994 (82.4% smaller)

The session results above ran on the older engine. Today’s engine is a little less aggressive on CSV and logs and far more aggressive on JSON (three JSON elision fixes landed 2026-09-30). The session benchmark has not been re-run on it. Without a query, the older engine gave 161 tokens for test output and 119 for YAML; today’s engine gives the same either way. File reduction is not session reduction. Every request also carries the system prompt, tool definitions, conversation, and caveman skill, and any recovery read counts. All five compressed files use S4 transforms, which need the local recovery store. Reproduce: generate the payloads with CorpusCases() from Caveman-Cloud internal/wrapbench/corpus.go, then call engine.New(nil, nil).Simulate(payload, engine.Options{Mode: engine.ModeCompress, Query: question}).

When Caveman wins

Long chatty outputs give terse style more removable prose. Measure your own A/B; no aggregate reduction is currently published.
Longer sessions can accumulate output reduction; measure the rules’ input cost and cache behavior in the same comparison.
Shorter replies can finish sooner and take less time to read.

When Caveman loses (net-negative)

The rules can add input tokens. Whether shorter output offsets that cost depends on the agent, cache behavior, workload, and billing model. Comparing raw input and output token counts alone does not establish a monetary net result.

Terse coding Q&A (#145): fixed prompt overhead can exceed any output reduction. The user in #145 measured a net loss.
Agents billed by request or credit (#506): GitHub Copilot charges premium requests. A shorter answer is the same request, so Caveman cannot lower Copilot credit use. Same applies to other per-message pricing.
Session totals can differ sharply from output-only changes because prompts, context, files, and injected rules consume tokens. Provider-billed A/B totals outrank output-only estimates.
Tool-side counters can go the wrong direction (#550). One Cursor A/B showed 4.3M tokens with caveman versus 1M without and twice the wall-clock time. The exact run was not reproducible, so the only safe conclusion is that rule re-injection, retries, and cache or context accounting can overwhelm output savings. Turn Caveman off if your A/B is net-negative.

Community A/B comparisons

Other people’s runs, not ours. Each names the skill version it tested. Caveman 3 rewrote the rules after these runs, so treat them as evidence about the approach, not about today’s exact file.

LemonCrow (#727): 20 engineering prompts × 5 reps × 3 runs (300 runs per arm), claude-opus-4-8, July 2026 skills/caveman/SKILL.md as the only extra system instruction. Billed output -44.1%. Total cost +0.06%. Initial cached context +1,532 tokens per run. Method · raw results
edubraqd (#733 comment, 2026-09-14): 10 dev questions in pt-BR, haiku-4-5, 20 replies per arm, pre-3.0 SKILL.md at the full level. On top of a CLAUDE.md that already asks for terse replies, caveman cut output 8% (median -5%, stdev 25%). Separately, across 662 Claude Code sessions, output was 0.3% of all tokens and caveman’s net effect on the bill was within ±0.2%.
JetBrains (blog, July 2026): 86 real coding tasks, paired A/B. 8.5% fewer output tokens, no measurable quality loss (p = 0.82).

Common thread: output shrink, but an agent run’s bill is mostly input and cache re-reads, which an output-style rule cannot touch. The proxy goes after that input instead.

Measure it yourself

/caveman-stats (Claude Code) reads the session log and prints recorded output/cache-read counts and mode attribution. It reports savings as unknown because that transcript has no measured comparison without Caveman. A benchmark on other tasks would not verify savings for this session.
Run the same task with and without Caveman, then compare provider usage or billing page. That A/B outranks repository estimates.
Reproduce repository numbers with benchmarks/run.py (Anthropic key required) and evals/measure.py (offline committed snapshot).

Rule of thumb

Compare provider-billed totals on the same task with and without Caveman. If Caveman increases billed cost for the same task, turn it off for that workload.

Earlier stats releases applied a fixed 65% output ratio without a committed reviewed result. Current reports ignore those historical est_saved_* fields while preserving the original history rows. Lifetime and shared reports do not convert those estimates into verified savings, and the statusline no longer displays the numeric savings suffix. Memory-file comparisons report original and current bytes; they do not prove either file was sent to a provider. If your A/B contradicts these numbers, open an issue. We will add the result to this page.

Testing and benchmarks in the repository

The repository uses language-specific tests plus cross-package verification. Run the smallest relevant command first, then broader gates before release.

Core test commands

Go:

go test ./...
go vet ./...
go build ./...

This covers Engine, proxy, memory, browser bridge, cache planner, and other Go packages in the module. TypeScript and JavaScript:

pnpm install --frozen-lockfile
pnpm -r build
pnpm -r test

Individual packages expose narrower scripts:

pnpm --dir packages/cli test
npm --prefix extension test
pnpm --dir packages/shared/contracts test

Use the package manifest to confirm exact scripts. Python:

python -m pytest -q packages/sdk/python

Repository verification:

python tests/verify_repo.py

The verifier checks repository invariants that ordinary unit tests do not cover, including generated or packaging expectations. Editing anything in src/hooks/ requires regenerating src/hooks/checksums.sha256, which the verifier compares digest by digest. Claude Code hooks:

npm test                                   # includes tests/hooks/*.test.mjs
node --test --test-force-exit tests/*.js
python -m unittest discover -s tests
bash tests/manual/session-mode-smoke.sh    # end-to-end, throwaway config dir

The smoke script drives the real hook binaries with the payloads Claude Code sends and never touches your own ~/.claude.

Platform coverage

Continuous integration builds and tests native Go and JavaScript paths and includes Windows coverage for platform-sensitive launcher behavior. The native release workflow builds a 36-artifact operating-system, architecture, and binary matrix, then publishes signed checksums. A local pass on one platform does not prove the full release matrix. Report exact commands and environment tested.

Benchmark categories

Engine fixtures: compare original and compact representations on committed fixtures. A valid report includes selected compressor, recovery result, byte or token counter, and any invariant checks.
Browser fixtures: measure accessibility-tree capture, query focus, compression, and recovery on recorded pages. See browse/BENCHMARK.md.
Cache corpus: stress capability and fail-safe gates. Report unsupported and rejected cases; raw pass rate is not a market-ranking result.
Subagent tax: packages/subagent-tax runs local fixtures without provider requests. Its counterfactual results apply to the exact fixture, assembly method, and token counter.
Wrap report: the published wrap benchmark documents one recorded comparison and provenance. See the CaveBench Wrap benchmark section above for reproduction availability and claim limits.

Writing a benchmark report

Every public result should include:

question being tested
fixture names, source, and hashes
code revision and date
hardware or provider/model where relevant
exact command
count basis and price source
baseline and treatment definitions
recovery, parse, and quality checks
failures, exclusions, and negative deltas
narrow conclusion supported by data

A fixture-level result is not production validation. A local token estimate is not provider cost, and successful retrieval does not establish quality parity.

Failure testing

Compression and integration tests should cover:

malformed input
duplicate or ambiguous JSON
empty and boundary-sized payloads
transformation larger than original
full or failed recovery store
unknown mode, provider, model, and grader
stream interruption
unsafe endpoint and redirect
secret-bearing headers
unsupported platform or upstream version

The expected safe result is often original input or explicit rejection. Tests should assert that behavior, not only successful compression.

⌘I

Get Started

Skills & Agents

Proxy & Engine

Build on Caveman

Reference

Caveman benchmarks and honest numbers

CaveBench Wrap benchmark

Per-case results

Method

Verification and provenance

Reproduction availability

Boundaries

Response skill measured numbers

Per-file compression

When Caveman wins

When Caveman loses (net-negative)

Community A/B comparisons

Measure it yourself

Rule of thumb

Testing and benchmarks in the repository

Core test commands

Platform coverage

Benchmark categories

Writing a benchmark report

Failure testing

​CaveBench Wrap benchmark

​Per-case results

​Method

​Verification and provenance

​Reproduction availability

​Boundaries

​Response skill measured numbers

​Per-file compression

​When Caveman wins

​When Caveman loses (net-negative)

​Community A/B comparisons

​Measure it yourself

​Rule of thumb

​Testing and benchmarks in the repository

​Core test commands

​Platform coverage

​Benchmark categories

​Writing a benchmark report

​Failure testing

CaveBench Wrap benchmark

Per-case results

Method

Verification and provenance

Reproduction availability

Boundaries

Response skill measured numbers

Per-file compression

When Caveman wins

When Caveman loses (net-negative)

Community A/B comparisons

Measure it yourself

Rule of thumb

Testing and benchmarks in the repository

Core test commands

Platform coverage

Benchmark categories

Writing a benchmark report

Failure testing