14 — The caveman ecosystem: flagship verified, cavemem and caveman-code deep-dived

Round-2 source sweep (2026-07-03; clones: caveman @ 25d22f8, caveman-code @ 74d599a, cavemem @ 1fe41e9). The flagship's hub claims were verified line-by-line against source; cavemem and caveman-code — previously one-line entries in the family diagram — get their first equal-depth analysis. Companion verification pass for RTK/headroom/lean-ctx/codedb: 15 — Verification pass.

TL;DR

• The flagship holds up. Every load-bearing claim in 01 — caveman design confirmed at source: six levels, two hooks, verbatim guards, Auto-Clarity, the eval methodology, the honest ~50%-over-terse-control number (re-run: +50% median). One wording fix: the installer hand-validates hook shapes to survive Claude Code's Zod, it doesn't run Zod itself. The README now states the 65% per-task mean on its own line — cite it directly. +6.8k stars in the 15 zero-commit days after v1.9.0 (2026-06-12) — the promotion-driven-adoption signal, re-confirmed. v1.9.1 shipped 2026-07-03 (50 changes: headline rebased to 65%, hook/installer hardening, docs/HONEST-NUMBERS.md — https://github.com/JuliusBrussee/caveman/releases/tag/v1.9.1), so the signal now rests on the star velocity alone, not on a frozen repo. First published null result exists (GrowwStacks: "no measurable token savings", via the servicesground roundup) alongside Pillitteri's 61–68%-on-discursive-text ≈ 4–10% whole-session — both consistent with the dossier's ~4–6% modeling.
• cavemem is refuted as shipped. Its compression is deterministic regex + a 5.9 KB lexicon, not the model; measured on its own eval corpus it delivers 10% (lite) / 33% (full, the default) / 36% (ultra) against the marketed 30/60/75 — and even those savings mostly never reach the model, because the MCP read tool re-expands by default, refunding abbreviations at read time. Compounding defects: FTS indexes compressed text but queries arrive raw (vocabulary mismatch breaks its own lexical retrieval), retrieval is global across every project on the machine, the summaries tier is unsearchable, excludePatterns privacy config is dead code, growth is unbounded, and hedge-stripping deterministically converts speculation into assertion ("It could potentially break prod" → "It break prod"). Dormant 2026-05-17 → 2026-07-03 (44 open issues, zero maintainer replies); maintenance resumed 2026-07-03 — issues triaged 44 → 4, npm dist/index.js packaging bug fixed, opt-in SSRF/ReDoS-hardened web-enrichment MCP tool added, Windows PowerShell 7 hooks, in-repo version 0.3.0 (npm still 0.2.1) — https://github.com/JuliusBrussee/cavemem/commits/main. The measurement-based refuted-as-shipped verdict stands until re-measured. One keepable artifact: the preserved-segment tokenizer (fence/code/URL/path/version/number spans byte-exact through any transform) — ~100 lines, the correct safety contract for prose transforms.
• caveman-code was mischaracterized. Not a "code variant" of the skill — a full terminal coding agent (open fork of pi-code, 9-package TS monorepo, TUI + CLI + SDK, 20+ providers) whose token story is four stacked layers: a 3-level register (~120–195 prompt tokens, hard exceptions for code/commit-messages/PR-text/security), per-tool head/tail line budgets ("Flint Chipper": bash 80, read 300, grep 120), read-dedup-by-fingerprint stubs, and RTK integration on-by-default when the binary is present — plus opt-in LLMLingua-2 (default off). Its "~2× fewer tokens vs Codex" is a real, artifact-backed benchmark (honest-bench: 524,703 vs 1,010,185 fresh tokens, 25 tasks, real child processes, committed CSV/JSON/logs) — but it measured the full stack at ultra+ML-on (more aggressive than shipped defaults), Codex's token accounting is asymmetric (printed totals may include cache), and caveman-code passed 1–2 fewer tasks than Codex in both runs. First-party-reproducible tier; the best benchmark hygiene in the family, with real caveats.
• Family growth: cavegemma (73★ 2026-07-03; 79★ 2026-07-05, still frozen since 2026-05-17) — a LoRA fine-tune of Gemma to "speak caveman natively", i.e. bake the register into weights and drop the prompt (self-host-only lever; watch, don't adopt). cavekit 1,068★ (2026-07-03); 1,074★ (2026-07-05), now at v4.1.0 (tag 2026-06-18, https://api.github.com/repos/JuliusBrussee/cavekit/tags). caveman-shrink is real and guarded (27k npm downloads/month at the 07-03 sweep; 23.3k/month as of 2026-07-05 — and its published tarball was missing spawn-options.js until fixed alongside v1.9.1, #597). The README now cites arXiv 2604.00025 ("+26 points under brevity") — the dossier's records ledger already marked that paper do-not-propagate (solo, unreviewed, cherry-picked subset).

Flagship verification (claims → verdicts)

cavemem (first deep dive)

Identity. TypeScript pnpm monorepo (7 packages + MCP server + CLI + embedding worker), MIT, created 2026-04-18, dormant 2026-05-17 → 2026-07-03 — the initial project lived one month, then went quiet; maintenance resumed 2026-07-03 (issues triaged 44 → 4, packaging fix, in-repo 0.3.0 — npm still 0.2.1; see the TL;DR). npm cavemem@0.2.1, ~2.6k downloads/month. 581★ / 1 watcher / 44 unanswered issues at the 07-03 sweep.

Architecture. SQLite WAL at ~/.cavemem/data.db: sessions, observations (every prompt + every tool call, 4 KB cap), summaries (turn/session), observations_fts (FTS5 porter), embeddings (raw Float32 blobs, no ANN index — brute-force cosine over all embeddings per query). Compression = code, not model: priority-resolved span tokenizer (fence > inline-code > URL > heading > path > date > version > number > identifier, byte-exact pass-through) + four regex drop-passes (pleasantries/hedges/fillers/articles) + a 5.9 KB abbreviation lexicon. expand() reverses abbreviations only. Retrieval: FTS5 BM25 + cosine, min-max + fixed-alpha blend (not RRF); embeddings computed on the expanded text (the one genuinely thoughtful design choice). MCP surface: 4 tools ≈ 300 tokens (genuinely lean). Five hooks capture prompts/tool-calls/turn-summaries; session-start injects up to 3 same-cwd prior session summaries.

Measured vs marketed. Marketing "~75%" is a config-schema description string ("lite ≈30%, full ≈60%, ultra ≈75%"), benchmarked nowhere. The repo's own spec targets 30/40/55. An evals/ bench exists (202-word corpus, chars/4 counter). Live run on that corpus: lite 10% / full 33% / ultra 36% — misses even its own targets. And get_observations defaults expand: true, so stored-compression savings are largely refunded before the model reads them; the at-rest savings mostly save disk, not context.

Failure inventory (each verified in source): FTS vocabulary mismatch (indexes compressed "db", queries raw "database" — porter can't bridge abbreviations); cross-project retrieval leakage (cwd-scoping parameter exists, no caller passes it); summaries tier unsearchable (no FTS, no embedding, no MCP tool); no dedup/TTL/eviction anywhere; no recency/supersede semantics (a stale April guess outranks a June correction); privacy.excludePatterns documented but referenced nowhere in executable code; hedge-stripping is deterministic modality destruction — every hedged memory is systematically overclaimed on recall, and expand can rewrite words the author wrote ("auth" → "authentication"); injection cost unmeasured and structurally uncapped (3 × 20-turn concatenated summaries at session start).

Corrections to the prior hub description: compression is not model-side; "lossy, no recovery" → partially reversible, semantically unsafe; "single-agent" → cross-IDE by design (Claude/Cursor/Gemini/OpenCode/Codex into one DB) but single-machine and unintentionally global-across-projects; "no benchmark" → trivial benchmark exists; "~75% (expect 55–60%)" → 33–36% measured, mostly refunded at read time; stats: 581★/1 watcher; dormant 7 weeks until maintenance resumed 2026-07-03 (see above).

Verdict. Architecturally superseded by any reversible-store design (headroom CCR / lean-ctx archive class): storing lossy prose with no ground-truth path makes corrupted memories unverifiable forever. The keepable artifact is the preserved-segment tokenizer as a transform safety contract, and the four self-inflicted wounds convert into requirements for any engine: index the canonical form (or transform queries identically), embed canonical text, scope retrieval per project by default, budget and measure injection.

caveman-code (first full analysis)

Identity. Standalone coding agent, TS monorepo of 9 packages, MIT, binaries caveman/caveman-code; an open, credited fork of pi-code with Claude Code-compatible config surfaces (~/.cave/settings.json, commands/, skills/, agents/*.md, .mcp.json; observer-only hooks). Plan mode, autonomous goal loop, architect/editor model split, session branching + shadow-git checkpoints, 5 builtin worktree-isolated subagents. Created 2026-04-08; 671★ / 3 watchers (the family's un-promoted member); npm @juliusbrussee/caveman-code 0.65.2, ~20.7k downloads/month, single published version.

The four compression layers (defaults: register full on, budgets on, dedup on, RTK on-when-present; LLMLingua-2 ML layer off):

1. Cave Mode register — 3 levels (no wenyan), 120–195 system-prompt tokens, hard exceptions at every level: code blocks, commit messages and PR descriptions, security warnings, destructive-op confirmations. Note the family's internal contradiction: the caveman plugin ships /caveman-commit that compresses commit messages; caveman-code exempts them — the same author reached the hub's own conclusion (order-sensitive/durable text should not be register-compressed).
2. Per-tool line budgets ("Flint Chipper") — head/tail caps per tool: bash 80 (50h/30t), read 300 (200/100), grep 120 (80/40), find/ls 60, fallback 150; + ANSI strip, blank-line collapse, semantic JSON/XML extraction. Claimed −67…94% per fixture (offline bench in-repo).
3. Read-dedup — per-session content fingerprints; re-reads return a stub ("−99% on repeats") — the sem/lean-ctx attention-ledger idea, independently converged on.
4. RTK bridge — detects the external Rust rtk binary and rewrites bash through rtk rewrite via a spawn hook, enabled by default: an agent vendor shipping RTK integration on-by-default is direct prior art for the layered-stack thesis.

The 2×-vs-Codex claim, audited. honest-bench (2026-05-18, committed harness + raw artifacts): 25 verify-scripted tasks, caveman and Codex CLI spawned as real child processes, same gpt-5.5/xhigh: 524,703 vs 1,010,185 fresh tokens (1.93×), 14/25 vs 15/25 resolved. A same-day thinking=high variant: 2.00× but −2 tasks. Caveats that must ride with the number: the caveman arm ran rtk=on, cave-mode=ultra, tool-compression=on, ml=on — more aggressive than shipped defaults; Codex tokens are stdout-scraped with a fallback path that books totals as input (cache_read=null) so the denominator may include cached tokens; single run per task, no CIs; pass rate below Codex in both runs. Tier: first-party but genuinely reproducible — the best benchmark hygiene in the family (config fingerprints in the CSV, per-task logs, one-command repro), well above the skill's README benchmark; still self-run and quality-negative.

Reusable for a native engine: the register exception list; the per-tool line-budget table (portable constants); read-dedup stubs; the subagent output contract ("file:line over prose, no emojis, terse"); and the honest-bench harness design (real processes, per-task verify.sh, fresh-token accounting, config-fingerprint column, committed artifacts) as the template for comparative benches.

What this changes in the hub

1. Family matrix rows updated: cavemem = refuted as shipped (33–36% real, refunded at read time; unmaintained); caveman-code = full agent, 4-layer stack, reproducible-but-caveated 1.93× — not a register variant.
2. The memory either/or table in 06 — Combining tilts further against cavemem: it now loses on measurement too, not just architecture.
3. "No agentic quality benchmark anywhere in the family" is obsolete — honest-bench exists and its pass-rate gap (−1/−2 tasks) is the first quality datapoint for register-style compression on agentic work: directionally negative, n too small to settle.
4. Evidence ledger additions: GrowwStacks null result; Pillitteri 4–10% whole-session; cavegemma as the register-in-weights frontier (self-host-only).