07 — Evidence, benchmarks, and the claim graveyard

This page is the skeptic's appendix to the comparison: what is actually measured versus self-reported for each tool, the one correction that applies to all four headlines, the consolidated graveyard of claims that do not survive a tokenizer check, and the runnable harness that converts any of these tools' marketed ratios into your own numbers. The governing rule throughout, inherited from the dossier: a per-payload compression ratio is not a banked dollar saving until it survives a validation harness on real tasks at equal quality.

The evidence-tier scheme

Every claim in this folder carries one of four tiers:

The one correction that applies to all four: per-payload ≠ whole-bill

All four tools advertise a big percentage, and all four percentages are the same kind of number — a best-case ratio on a favorable payload — not a share of the dollar bill. The correction is identical in each case:

   THE HEADLINE                    THE WHOLE-BILL REALITY
   ────────────                    ──────────────────────
   caveman:  "~75%"        ──►  per-payload on visible prose; prose is ~17% of
                                $; realistic whole-bill ~4–6%/day, hard cap 17%

   headroom: "60–95%"      ──►  per-payload on redundant logs/JSON; code & grep
                                compress 0%; production median 4.8% whole-session

   RTK:      "60–90%"      ──►  per-command on verbose commands; only Bash output;
                                whole-bill = Bash-share × compression × (write+0.1×read)

   lean-ctx: "up to 99%"   ──►  per-read on CODE (map/signatures 96–99%) or the
                                ~13-tok cache-handle; prose/config compress <10%;
                                "86% session" is a code-read-heavy best case

Two structural facts drive the gap between headline and bill, and they apply to every input tool (RTK, headroom, lean-ctx) equally:

1. Most input tokens already read at 0.1×. After a token is first written to the cache, every later read costs a tenth of input price. So compressing a token that will mostly be read saves a tenth of its face value. The dollar win is concentrated on the first write, not the many reads.
2. None of the four touches thinking (20% of dollars). The headline percentage, whatever it is, applies to at most the output bucket (caveman, 17%) or the compressible-observation slice of the input bucket (RTK/headroom/lean-ctx, part of 61%) — never to the whole bill.

The result: even an aggressive deployment of any one of these lands in the low double digits of dollars at best, not at its headline percentage. That is still a real lever on a real bucket; it is simply not the marketed number.

Per-tool benchmarks: what is real, what is self-report

Caveman

Headroom

RTK

The underlying levers RTK uses are nonetheless T1 (locally reproduced in the dossier): the log filter at −94.2%, JSON minify at −34.3%. RTK's mechanism is T1; RTK's specific product numbers are T4.

lean-ctx

All figures below are locally reproduced in this round by building lean-ctx v3.8.9 from source and running lean-ctx benchmark report . on the lean-ctx repo (tiktoken o200k_base, 50 files / ~479K raw tokens) and on individual reads — the same "build it and measure" tier the dossier applies to caveman.

lean-ctx's mechanism is T1 (the read modes, shell compression, and BM25/graph search all reproduce here); its specific product percentages are T4 (self-measured, GPT tokenizer, per-read/per-session best cases, no independent replication). Its asymmetry mirrors headroom's inverted: where headroom compresses logs/JSON and passes code through, lean-ctx crushes code and barely touches prose/config — so the "right" headline depends entirely on whether your reads are source code or documents.

Adoption stats — and why to ignore them

Three repos carry large, PR-inflated star counts with abnormally low watcher ratios; lean-ctx is the youngest and least-inflated but has no external evidence. In this niche stars are a marketing artifact, not adoption; rank by evidence, forks, and issue activity instead.

A healthy repo's star:watcher ratio is roughly an order of magnitude lower. The takeaway is uniform: do not read these star counts as proof of quality or adoption. Headroom is the best externally instrumented (production telemetry + one independent measurement); caveman is the most transparent (the mechanism is a readable prompt); lean-ctx is the best self-instrumented (bounce-netted, signed ledger, 2,900+ tests) but the least externally verified; RTK is the least verified of all.

The consolidated claim graveyard

Every popular-but-overstated claim about the three, with the corrected reading, in one table.

The validation harness

This is how to turn any of the three (or the stack) from a marketed ratio into a banked, quality-verified saving on your own workload. It is a paired-task benchmark with cache continuity and command-re-run rate as first-class metrics.

Arms (run the same fixed task suite through each, fresh sessions, same effort):

Metrics (read from session JSONL usage fields):

• tool-result tokens, and total tokens per solved task (not per task);
• cache_read ratio and cache-write spikes — the make-or-break for any input compressor;
• command re-run / bounce rate — did the agent re-run a command or re-read a file in full after a compressed read? (the dropped-context tell for RTK, and exactly what lean-ctx's adjusted_total_saved already nets out — cross-check its self-report against your own count);
• fraction of observation tokens that flow through Bash at all (bounds what RTK can even touch);
• retrieve count and retrieve token cost (headroom CCR headroom_retrieve, lean-ctx ctx_expand);
• task success / tests pass (objective where possible); wall-clock.

Canary tasks targeting known compression failure modes: negation preservation ("don't do X"), ordering-sensitive instructions, numeric precision, and a detail buried in a payload that compression is likely to drop (to test whether the model knows to retrieve, for headroom, or re-runs, for RTK).

Acceptance rule (per tool, versus the appropriate baseline arm):

Accept a compressor for token optimization only if, versus the baseline arm:
  task / test success           >= baseline
  cache_read ratio              >= baseline   (no silent cache-bust)
  command re-run rate           <= baseline   (no silent dropped-context cost; RTK)
  total tokens per solved task  <= baseline by at least 20%
  net of the tool's own overhead (MCP schema rent, retrieve round-trips,
  hook-registration / host-write, ~200-500 tok proxy metadata)

For RTK specifically, A/B against the Hooks arm (a hand-written filter), not the Native arm — RTK earns its place only if its 100+-command coverage beats a filter you could write yourself, net of the dropped-context risk and the hook-conflict surface with caveman.

Source ledger

The full consolidated source ledger (every citation, with access dates), the formal per-technique records (C1 / H1–H4 / R1 / L1), and the unverified-claims register now live in 08 — Records, ledger & unverified — the hub's single complete reference. Key sources, summarized:

• Caveman — repo JuliusBrussee/caveman; the family records, the tokenizer measurement battery, and the folklore ledger: dossier 03 — prior-art and market scan and 10 — style and language compression.
• Headroom — repo chopratejas/headroom; companion model chopratejas/kompress-base; the source audit, benchmark tables, H1–H4 records, and headroom-specific graveyard: dossier 53 — headroom and context compression.
• RTK — repo rtk-ai/rtk; the architecture doc, the per-command tables, the integration matrix, and the RTK-specific graveyard: dossier 56 — RTK and write-time observation compression.
• lean-ctx — repo yvgude/lean-ctx (v3.8.9); site leanctx.com (compare/pricing); ARCHITECTURE.md, BENCHMARKS.md, LEANCTX_FEATURE_CATALOG.md; locally built + benchmarked this round. The full L1 record, source audit, and lean-ctx graveyard live in the records page and the design teardown.
• The compression market and cache-safety classification (including the published RTK-vs-headroom head-to-head, the independent headroom measurements, and the "rank by evidence not stars" rule): dossier 54 — context-compression literature and market.
• The structural alternative RTK and headroom are not (persistent symbol index): dossier 51 — code-intelligence tools.
• The economics and the 10× verdict these percentages sit inside: dossier index and 00 — executive summary.
• The container-adoption hazards (host-write ban, hook reconciliation, role-scoping): architect code-intelligence tooling roadmap.

Next: 08 — Records, ledger & unverified for the formal per-technique records and the full source ledger. Back to the overview, or up to the token-optimization dossier for the surrounding economics.