--- title: Logs summary: Keep every error and the lines around it, fold the thousand identical INFO lines. canonical: https://docs.caveman.so/docs/proxy/compressors/logs layer: proxy license: BSL-1.1 capability: engine updated: 2026-09-16T21:32:40-07:00 basis: inferred --- # Logs > Keep every error and the lines around it, fold the thousand identical INFO lines. The log compressor keeps every error, warning and stack frame verbatim, keeps the head and tail of the stream, and folds each run of routine lines into one marker that states how many lines it replaced and what they had in common. It is line oriented: lines keep their order and their text. ## Before and after The fixture is 302 lines of application log: one INFO line per second, one ERROR with two stack frames, one WARN. ```text 2026-03-04T09:12:01Z INFO checkout worker=0 order=ord-3200 status=ok latency_ms=40 2026-03-04T09:12:02Z INFO checkout worker=1 order=ord-3201 status=ok latency_ms=41 2026-03-04T09:12:03Z INFO checkout worker=2 order=ord-3202 status=ok latency_ms=42 ``` ```bash curl -O https://docs.caveman.so/examples/compressors/logs/app.log caveman-engine compress < app.log 2> report.json ``` The whole output, 12 lines: ```text 2026-03-04T09:12:01Z INFO checkout worker=0 order=ord-3200 status=ok latency_ms=40 2026-03-04T09:12:02Z INFO checkout worker=1 order=ord-3201 status=ok latency_ms=41 … 140 lines elided (caveman): all status=ok; order: ord-3202..ord-3341 all 140 present; range worker=0..7 latency_ms=40..64 … 2026-03-04T09:14:23Z ERROR checkout worker=7 order=ord-3391 msg="payment gateway timeout after 30s" at billing.charge(billing.go:214) at checkout.submit(checkout.go:88) … 68 lines elided (caveman): all status=ok; order: ord-3343..ord-3410 all 68 present; range worker=0..7 latency_ms=40..64 … 2026-03-04T09:15:32Z WARN cache worker=2 msg="redis latency 480ms above threshold" … 86 lines elided (caveman): all status=ok; order: ord-3412..ord-3497 all 86 present; range worker=0..7 latency_ms=40..64 … 2026-03-04T09:16:59Z INFO checkout worker=2 order=ord-3498 status=ok latency_ms=63 2026-03-04T09:17:00Z INFO checkout worker=3 order=ord-3499 status=ok latency_ms=64 … caveman: elided lines resemble shown lines and match the stated invariants — answer from this view; if you truly cannot, one broad caveman_retrieve … ``` The report on stderr, with the per-run digests and the handle trimmed to `…`: ```json {"content_type":"log","tokens_before":9918,"tokens_after":385,"ratio":0.961181689856826, "basis":"inferred","recovery_handle":"ccr_773d…","method":"log"} ``` Captured from a caveman-engine build dated 2026-08-24; `caveman setup --install` today pins bin-v1.1.7. 9,918 tokens to 385, counted by the offline `o200k_base` counter behind [Token counting](/docs/proxy/tokens). ## How it works Input that is not valid UTF-8 is reported as a parse problem and the caller forwards the original bytes. So is anything under 4 lines. Lines are split on LF with each line keeping its own CR, so a file with mixed endings rejoins byte for byte, and any marker the compressor writes takes the ending the majority of the file uses. Each line is then marked keep or drop. The first 2 lines are kept. The tail window then walks backwards keeping lines until it has kept 2 real log lines; a marker or contract line it passes is kept too but does not use up the budget, so compressing an already compressed view yields the same bytes again. A line is kept when it matches the importance pattern: the words `ERROR`, `FATAL`, `PANIC`, `EXCEPTION`, `TRACEBACK`, `FAIL`, `FAILED`, `FAILURE`, `WARN` or `WARNING` at a word boundary, case insensitive, or a line starting with indented `at `, or with `File "`, or containing `.go:` followed by digits, or starting with an indented `--->`, or containing `caused by`. An ASCII line that starts with a visible character is first scanned for those literals, which is a fast path with the same result as the pattern. A line that is already an elision marker or the contract line is kept, which is what makes a second pass produce the same output as the first. When a query is supplied, deterministic BM25 scores every line, and up to 16 lines scoring at least 0.30 of the best score are added. The redundancy guard adds one representative of every distinct kind of line, so a dropped line always resembles a line still visible. It compares each line by its set of words with digit runs masked to `#`, and treats a line as represented when 90 percent of its vocabulary appears in one surviving line. After 128 promotions the payload is treated as a document and the remaining lines are all kept, which makes the output no smaller and sends the original through untouched. The masking rules, and the cap on how many kept lines it compares against, are on [Compressors](/docs/proxy/compressors#what-every-compressor-guarantees). Kept lines are written in order. Each run of dropped lines becomes one line: `… N lines elided (caveman): …`, or `… N lines elided (caveman) …` when the run has no summary. A run of fewer than 3 lines is only collapsed when it has a summary and the marker costs at most half the bytes it replaces; otherwise the lines themselves are written out. When the payload dropped at least 4,096 bytes and does not already carry one, a single contract line is appended telling the reader how to read the view. It is added once per payload. ### What the summary can say Fields come from the line itself: a line that is a single JSON object is read as one, otherwise the `key=value` pairs are taken in order, quoted values unquoted, up to 24 fields per line. The summary is then computed across exactly the lines of that one marker, under a budget of 160 bytes that is also capped at a quarter of the bytes replaced, with a floor of 96. | Fact | Form | Condition | | --- | --- | --- | | Constant | `all status=ok` | Byte-identical in every elided line. | | Enumeration | `status: fulfilled×50 shipped×6 processing×4` | At most 5 distinct values including an `absent` bucket, each at most 24 bytes with no spaces. The counts sum to the elided count. | | Range | `range latency_ms=40..64` | Present in every line, and every value parses as a number. The bounds are the original value strings. | | Coverage | `order: ord-3202..ord-3341 all 140 present` | Identifier-like values. The `all N present` form is printed only after counting a gapless fixed-width integer range behind a shared prefix. At most 2 coverage entries. | Fields whose name matches `secret|passw|token|api[_-]?key|authoriz|credential|cookie|session` are left out, as are values over 40 bytes. An entry the budget cannot fit is dropped whole and the summary ends with `+N omitted`. ## What is always kept The first two lines and the last two real lines. Every line matching the importance pattern, wherever it sits, including every stack frame. One representative of every distinct kind of line. Every marker from an earlier pass. Line order, line text, and the line endings the file used. ## When it is chosen Detection calls a payload `log` when it has at least 4 lines and at least half of them, and no fewer than 3, carry a level token or a timestamp: `TRACE|DEBUG|INFO|WARN|WARNING|ERROR|FATAL|PANIC` at a word boundary, a leading `[LEVEL]`, an ISO timestamp such as `2026-03-04T09:12:01`, or a bare `HH:MM:SS`. ```bash caveman-engine detect < app.log ``` ```text log ``` See [how detection decides](/docs/proxy/compressors#how-detection-decides) for where this test sits among the others. Both floors are the same 4 lines: detection needs 4, and the compressor declines under 4. It also declines on invalid UTF-8. When nothing was dropped the output equals the input, and the engine's own check forwards the original with ratio 0. ## Options A query, when the caller supplies one, adds up to 16 relevance-selected lines. An empty query behaves exactly like a queryless run, and a query keeps the output at most the size of the queryless view. ## Recovery The original bytes are stored before the compressed view is emitted, and the handle in the report retrieves them: see [Recovery](/docs/proxy/recoverable).