Config files

The config compressor handles YAML, TOML, and INI. It keeps every top-level YAML key, every line naming a secret or a failure, the head and tail of the file, and the parents and children of anything it kept, so the YAML it emits still parses. Dropped lines become one comment line stating how many went.

Before and after#

The fixture is a 159 line service manifest: a service block with a database section, fifty near-identical feature_NN blocks, and an incident block at the end.

service:
  name: checkout
  owner: payments
  database:
    host: db.internal
    port: 5432
  feature_00:
    enabled: true
    rollout: 0

curl -O https://docs.caveman.so/examples/compressors/config/service.yaml
caveman-engine compress < service.yaml 2> report.json

Stdout is the whole compressed file, 16 lines. This is all of it.

service:
  name: checkout
  owner: payments
  database:
    host: db.internal
    port: 5432
  feature_00:
    enabled: true
    rollout: 0
  feature_01:
    enabled: false
    rollout: 1
# … 144 config lines elided (caveman) …
  incident:
    status: ERROR
    action: rollback

The report on stderr carries the counts. The handle varies per run and per machine, so it is trimmed with … here.

{"content_type":"config","tokens_before":841,"tokens_after":87,"ratio":0.896551724137931,
 "basis":"inferred","recovery_handle":"ccr_d83c…","method":"config","lossless_to_model":false}

Captured from a caveman-engine build dated 2026-08-24; caveman setup --install today pins bin-v1.1.7.

841 tokens to 87, counted by the offline o200k_base counter described on Token counting. The incident block survives because ERROR and ROLLBACK are both on the important-word list, and incident: itself survives as their parent.

TOML and INI take the same path. A 130 line file of twenty [pool.shard_NN] sections went from 651 tokens to 95, keeping [server], the first shard section in full, and [audit] with its on_write_failure = ROLLBACK line.

[server]
host = 0.0.0.0
port = 8080
workers = 8
timeout_ms = 30000

[pool.shard_00]
driver = postgres
max_conns = 40
idle_conns = 8
weight = 1

# … 114 config lines elided (caveman) …
[audit]
retention_days = 365
on_write_failure = ROLLBACK

How it works#

parseConfig classifies the payload first. It walks every line, skipping blanks and lines starting with # or ;, and counts four kinds:

Those four counts added together are the file's significant lines. A blank line, a line starting with # or ;, and a - sequence item are none of the four and count towards none of them.

The payload is TOML or INI when it has at least 2 sections, at least 4 assignments, zero prose lines, and no """ or ''' anywhere. It is YAML when it has at least 6 YAML keys, those keys are at least half of all significant lines, prose is at most a quarter of them, no line ends in a | or > block scalar header, and yaml.v3 decodes the whole payload. Anything else is not config and the compressor declines.

A classified file of at least 14 lines is then marked line by line:

1. 1

   The first 3 and the last 2 lines.

2. 2

   Any line matching ERROR|FAIL|FAILED|FATAL|PANIC|EXCEPTION|WARNING|SECURITY|SECRET|TOKEN|PASSWORD|DENIED|REJECTED|ROLLBACK, case insensitive, on a word boundary.

3. 3

   In YAML, every line at indentation 0 that is a key. A tab counts as two columns.

4. 4

   Query relevance, when a query is supplied. Deterministic BM25 scores the lines, anything at or above 0.30 of the best score becomes a candidate, and candidates fill up to 16 extra lines in score order.

5. 5

   One representative of every distinct kind of line, added by the shared redundancy guard, so a dropped line always resembles a line still visible.

Then the structure around the kept lines is restored, which is what keeps the output parseable.

In YAML, each kept line walks backwards for the nearest earlier line at a smaller indentation that is a parent (a key ending in :, or a - item), keeps it, and repeats from that line's indentation up to column 0. A second pass then walks forward from every kept parent and keeps up to 8 following lines that are indented deeper than it, stopping at the first line indented the same or less. That second pass is why a query that matches one nested value brings its sibling values with it.

In TOML and INI, every kept line pulls in its enclosing [section] header, and every kept section header pulls in up to 8 following lines, stopping at the next section header.

Each run of dropped lines becomes one comment line, # … N config lines elided (caveman) …, carrying the same line ending as the file. The output is rejected if it is no shorter than the input, if it kept every line, or, for YAML, if yaml.v3 no longer decodes it.

What is always kept#

The first three and last two lines. Every top-level YAML key. Every line naming an error, a failure, a secret, a token, a password, or a rollback. The parent chain above every kept line and up to 8 lines under every kept parent or section. One line of each distinct shape. YAML output always parses; it is checked before the result is returned.

When it is chosen#

LooksConfig runs the same classifier and returns true when it succeeds and the file has at least 12 lines. See how detection decides for where this test sits among the others.

caveman-engine detect < service.yaml

config

The compressor declines, and the engine forwards the original bytes, when the payload already contains a config lines elided (caveman) marker, when the classifier rejects it, when the file is shorter than 14 lines, when every line was marked to keep, when the result is no shorter, when compressed YAML no longer decodes, and when no recovery store is available.

Options#

None. NewConfig fixes the floor at 14 lines, 3 head lines, 2 tail lines, a query fill of 16 lines, a relevance threshold of 0.30, and the 8 line context window on both expansion passes.

Recovery#

The original file is stored before the compressed view is emitted, and the handle in the report retrieves it byte for byte: see Recovery.