Skip to main content
Caveman labels every number by how it was produced. Token estimates, provider usage records, benchmark comparisons, and verified savings records each answer different questions, and they must not share one label. This page explains the evidence bases, how token counts and costs are calculated, and how to read the caveman stats report.

​
Evidence bases

UI and reports should show the basis next to the value. A value cannot silently change basis while being aggregated.

​
Token counts

The Engine uses offline o200k_base counting where available and character estimate fallback. Those counts are inferred for provider billing purposes. Providers may tokenize the same text differently and may count cache or image inputs under separate units. Provider-reported token fields keep the provider basis. Do not replace them with a local estimate when the field is missing.

​
Cost calculation

The provider catalog contains dated public list prices. Local cost display is:
This is not an invoice. It can differ because of negotiated terms, subscriptions, regional prices, batch rates, caching rules, credits, taxes, or provider billing adjustments. Unknown provider or model prices produce zero plus unpriced. Zero prevents invented cost from entering totals; unpriced prevents zero from being mistaken for free use.

​
Inferred headroom

Local compression and learning reports can estimate avoidable context, reported as headroom under stated assumptions. Keep its time basis unchanged: a per-day estimate must not become a monthly claim unless the method measures each day and states that the result is a projection.

​
Benchmark evidence

A useful benchmark report states:
  • exact fixtures and their hashes or versions
  • command and code revision
  • counter or provider usage source
  • baseline and variant definitions
  • quality or invariant checks
  • failures and excluded cases
  • scope of conclusion
One fixture result supports only that fixture and method. Average reduction does not prove equal task quality, and recovery checks establish source availability rather than model comprehension.

​
Verified values

verified is reserved for methods whose prerequisites are enforced and whose records identify the method. Public local runtime should not mark Engine estimates, skill output, pixel conversion, TOON output, cache plans, or merged code as verified by themselves. When evidence is incomplete, use inferred, observed, provider_reported, or honest zero.

​
Negative and failed results

Keep negative deltas and failed transformations. Dropping regressions biases a result. A rejected optimization can still incur provider usage; account for that usage even though compact output was not served.

​
Publication checklist

Before publishing a number:
  1. Name the exact evidence basis.
  2. Link the committed fixture or source record.
  3. Disclose the counter and pricing date.
  4. State the quality test and failure count.
  5. Distinguish list price from invoice.
  6. Avoid extrapolating across model, provider, task, or time.
  7. Publish zero or unpriced when support is absent.
See Honest Numbers for currently supported public claims.

​
Local stats and savings accounting

caveman stats reports the requests routed through this machine’s Caveman proxy. It prints a terminal summary and writes an offline dashboard. In an interactive terminal the dashboard opens automatically; --plain keeps the command in the terminal. --json produces the complete report without opening a browser or writing HTML.
The default window is 30 UTC calendar days. Provider, model, agent, and auth filters use the exact values captured on each request. A current model selection never changes the price or identity of older requests. The report does not scan prompts from other agents or assume their traffic used Caveman.

​
What a token delta means

For an eligible request, the proxy normalizes original and final request JSON and counts both with the same offline tokenizer. Key order and equivalent escaping cannot manufacture a reduction. The difference includes replacement text, recovery markers, and request overhead. Negative differences remain negative. Original-request fallbacks earn no compression savings. Failed and incomplete requests do not earn savings merely because a compressor produced output. These are local estimates, not exact provider token counts. The provider can use a different tokenizer, add hidden framing, or bill image and audio content separately. The report keeps provider-reported usage apart from the local before/after pair. Multimodal requests, payloads over 4 MiB, and unsupported comparisons remain unavailable instead of counting base64 image bytes as text tokens. The baseline is the request at the proxy boundary. It answers what this request would contain without the proxy’s transforms. It does not reconstruct an entire alternate agent run. Native masking or MCP compression performed before that boundary, extra agent turns, recovery behavior, output verbosity, and task quality need their own paired evidence. A file is only represented when its read output reaches this boundary; a file’s size on disk alone earns no savings. Repeated requests are repeated observations, not unique files.

​
Three separate money questions

  • API spend: provider-counted billing buckets at the captured published rates. Complete reported usage remains eligible when a stream later fails; that failure still earns no counterfactual savings. It is a catalog estimate, not an invoice reconciliation.
  • Estimated API input reduction: the signed input delta valued using the request’s observed mix of uncached input, cache reads, and cache writes. The original request’s cache behavior is unknown, so this remains an estimate.
  • Subscription API equivalent: the same arithmetic for subscription or non-billable OAuth traffic. It illustrates the API value of that context; it does not reduce a subscription bill or prove additional quota.
OAuth is an authentication mechanism, not a billing plan. Paid cloud OAuth routes such as Vertex stay in the API category when their billing provenance supports it. Unknown billing modes stay separate. Unknown models never borrow a sibling model’s rate or a blended provider price. Rates and catalog provenance are captured with new requests. Prices are not recomputed from the model selected when the report is opened. Unsupported prices and absent measurements are null/unavailable; a displayed subtotal only covers the priced rows. Legacy compressed-segment measurements remain separate from the stronger request comparison. The report never promotes local evidence to verified savings or projects a window into a monthly claim.