Skip to content
Cavemandocs
Open Caveman
01Proxy/BSL-1.1

Logs

Keep every error and the lines around it, fold the thousand identical INFO lines.

View as Markdown

The log compressor keeps every error, warning and stack frame verbatim, keeps the head and tail of the stream, and folds each run of routine lines into one marker that states how many lines it replaced and what they had in common. It is line oriented: lines keep their order and their text.

Before and after#

The fixture is 302 lines of application log: one INFO line per second, one ERROR with two stack frames, one WARN.

text
2026-03-04T09:12:01Z INFO  checkout worker=0 order=ord-3200 status=ok latency_ms=40
2026-03-04T09:12:02Z INFO  checkout worker=1 order=ord-3201 status=ok latency_ms=41
2026-03-04T09:12:03Z INFO  checkout worker=2 order=ord-3202 status=ok latency_ms=42
terminal
curl -O https://docs.caveman.so/examples/compressors/logs/app.log
caveman-engine compress < app.log 2> report.json

The whole output, 12 lines:

text
2026-03-04T09:12:01Z INFO  checkout worker=0 order=ord-3200 status=ok latency_ms=40
2026-03-04T09:12:02Z INFO  checkout worker=1 order=ord-3201 status=ok latency_ms=41
… 140 lines elided (caveman): all status=ok; order: ord-3202..ord-3341 all 140 present; range worker=0..7 latency_ms=40..64 …
2026-03-04T09:14:23Z ERROR checkout worker=7 order=ord-3391 msg="payment gateway timeout after 30s"
  at billing.charge(billing.go:214)
  at checkout.submit(checkout.go:88)
… 68 lines elided (caveman): all status=ok; order: ord-3343..ord-3410 all 68 present; range worker=0..7 latency_ms=40..64 …
2026-03-04T09:15:32Z WARN  cache worker=2 msg="redis latency 480ms above threshold"
… 86 lines elided (caveman): all status=ok; order: ord-3412..ord-3497 all 86 present; range worker=0..7 latency_ms=40..64 …
2026-03-04T09:16:59Z INFO  checkout worker=2 order=ord-3498 status=ok latency_ms=63
2026-03-04T09:17:00Z INFO  checkout worker=3 order=ord-3499 status=ok latency_ms=64
… caveman: elided lines resemble shown lines and match the stated invariants — answer from this view; if you truly cannot, one broad caveman_retrieve …

The report on stderr, with the per-run digests and the handle trimmed to …:

json
{"content_type":"log","tokens_before":9918,"tokens_after":385,"ratio":0.961181689856826,
 "basis":"inferred","recovery_handle":"ccr_773d…","method":"log"}

Captured from a caveman-engine build dated 2026-08-24; caveman setup --install today pins bin-v1.1.7.

9,918 tokens to 385, counted by the offline o200k_base counter behind Token counting.

How it works#

Input that is not valid UTF-8 is reported as a parse problem and the caller forwards the original bytes. So is anything under 4 lines. Lines are split on LF with each line keeping its own CR, so a file with mixed endings rejoins byte for byte, and any marker the compressor writes takes the ending the majority of the file uses.

Each line is then marked keep or drop.

  1. The first 2 lines are kept. The tail window then walks backwards keeping lines until it has kept 2 real log lines; a marker or contract line it passes is kept too but does not use up the budget, so compressing an already compressed view yields the same bytes again.

  2. A line is kept when it matches the importance pattern: the words ERROR, FATAL, PANIC, EXCEPTION, TRACEBACK, FAIL, FAILED, FAILURE, WARN or WARNING at a word boundary, case insensitive, or a line starting with indented at , or with File ", or containing .go: followed by digits, or starting with an indented --->, or containing caused by. An ASCII line that starts with a visible character is first scanned for those literals, which is a fast path with the same result as the pattern.

  3. A line that is already an elision marker or the contract line is kept, which is what makes a second pass produce the same output as the first.

  4. When a query is supplied, deterministic BM25 scores every line, and up to 16 lines scoring at least 0.30 of the best score are added.

  5. The redundancy guard adds one representative of every distinct kind of line, so a dropped line always resembles a line still visible. It compares each line by its set of words with digit runs masked to #, and treats a line as represented when 90 percent of its vocabulary appears in one surviving line. After 128 promotions the payload is treated as a document and the remaining lines are all kept, which makes the output no smaller and sends the original through untouched. The masking rules, and the cap on how many kept lines it compares against, are on Compressors.

Kept lines are written in order. Each run of dropped lines becomes one line: … N lines elided (caveman): <summary> …, or … N lines elided (caveman) … when the run has no summary. A run of fewer than 3 lines is only collapsed when it has a summary and the marker costs at most half the bytes it replaces; otherwise the lines themselves are written out.

When the payload dropped at least 4,096 bytes and does not already carry one, a single contract line is appended telling the reader how to read the view. It is added once per payload.

What the summary can say#

Fields come from the line itself: a line that is a single JSON object is read as one, otherwise the key=value pairs are taken in order, quoted values unquoted, up to 24 fields per line. The summary is then computed across exactly the lines of that one marker, under a budget of 160 bytes that is also capped at a quarter of the bytes replaced, with a floor of 96.

FactFormCondition
Constantall status=okByte-identical in every elided line.
Enumerationstatus: fulfilled×50 shipped×6 processing×4At most 5 distinct values including an absent bucket, each at most 24 bytes with no spaces. The counts sum to the elided count.
Rangerange latency_ms=40..64Present in every line, and every value parses as a number. The bounds are the original value strings.
Coverageorder: ord-3202..ord-3341 all 140 presentIdentifier-like values. The all N present form is printed only after counting a gapless fixed-width integer range behind a shared prefix. At most 2 coverage entries.

Fields whose name matches secret|passw|token|api[_-]?key|authoriz|credential|cookie|session are left out, as are values over 40 bytes. An entry the budget cannot fit is dropped whole and the summary ends with +N omitted.

What is always kept#

The first two lines and the last two real lines. Every line matching the importance pattern, wherever it sits, including every stack frame. One representative of every distinct kind of line. Every marker from an earlier pass. Line order, line text, and the line endings the file used.

When it is chosen#

Detection calls a payload log when it has at least 4 lines and at least half of them, and no fewer than 3, carry a level token or a timestamp: TRACE|DEBUG|INFO|WARN|WARNING|ERROR|FATAL|PANIC at a word boundary, a leading [LEVEL], an ISO timestamp such as 2026-03-04T09:12:01, or a bare HH:MM:SS.

terminal
caveman-engine detect < app.log
text
log

See how detection decides for where this test sits among the others.

Both floors are the same 4 lines: detection needs 4, and the compressor declines under 4. It also declines on invalid UTF-8. When nothing was dropped the output equals the input, and the engine's own check forwards the original with ratio 0.

Options#

A query, when the caller supplies one, adds up to 16 relevance-selected lines. An empty query behaves exactly like a queryless run, and a query keeps the output at most the size of the queryless view.

Recovery#

The original bytes are stored before the compressed view is emitted, and the handle in the report retrieves them: see Recovery.