# 13 — Reuse surfaces and deltas (2026-07-03 source sweep) (https://jackin.tailrocks.com/reference/research/token-optimization-tools/13-reuse-surfaces-and-deltas/) # 13 — Reuse surfaces and deltas (2026-07-03 source sweep) [#13--reuse-surfaces-and-deltas-2026-07-03-source-sweep] This page records the 2026-07-03 re-sweep of the hub's four tools — fresh clones, registry checks, release deltas since the 06-15/06-18/06-20 snapshots — through a new lens the hub had not applied: **the Rust reuse surface**. The motivating question comes from the jackin❯ side: instead of *running* these tools in a container, which of their mechanisms can be **embedded natively** (crate dependency, vendored corpus, or ported kernel) into one engine? The resulting product design lives in the [jackin❯ Context Engine spec](/reference/research/jackin-context-engine/); two tools new to the research (sem, mq) are covered in [hub 25 — sem and mq](/reference/research/token-optimization-tools/25-sem-and-mq-structural-and-markdown-retrieval/). Everything below is source-verified with the same T1–T4 discipline; corrections to earlier hub pages are consolidated at the end. ## TL;DR [#tldr] * **RTK** (v0.43.0; develop @ `b52e02b`): still not a library, but its **highest-value assets are cleanly vendorable** — 64 declarative TOML filters with inline tests, an 85-rule command classifier + shell lexer with an unattestable-construct guard, and two new safety contracts worth adopting anywhere: the v0.43 **`never_worse` guard** (filtered ≤ raw or emit raw) and three-tier `ParseResult` degradation (`Full/Degraded/Passthrough`, "never return false data silently"). The hub's "12 strategies" was a *taxonomy*; the real dispatch surface is 42 native command modules + 64 TOML filters + 85 rewrite rules. * **headroom** (v0.28.0; moved `chopratejas` → `headroomlabs-ai`, Apache-2.0 unchanged, telemetry now opt-in): a phased **Python→Rust rewrite is \~⅔ executed** — the typed-compressor hot path is genuinely Rust today (PyO3 inside the Python proxy) — but nothing is on crates.io and the crates are parity-constrained 0.1.0. The **ML stage is being demoted by upstream itself** (timeout cascades; deterministic `TextCrusher` replacement), vindicating the hub's caution. The live-zone algorithm, CCR schema (blake3/24-hex, `<>`), cache-economics gate (`net_mutation_gain`, 1.25×/0.1×), and auth-mode policy are now **documented to reimplementation grade**. * **lean-ctx** (v3.8.18): now **published on crates.io** — but with explicitly no internal-API stability, and the curated SDK facade is `publish = false` git-only, pinning tree-sitter×26 + ort + http into any consumer. **The hub's "no independent benchmark" is obsolete, unflatteringly**: the pre-registered tokbench R1 ran lean-ctx at defaults and found a \~3K-token injected-prefix tax re-billed every turn and `102 native bash / 0 ctx_shell` (routing never engaged). Upstream answered with a "faithful arm" minimal profile. Lesson elevated to a spec invariant: **savings must be reported net-of-injection**. Portable kernels: read-stub cache, deterministic crushers, `signatures_ts`, CFT knapsack compiler, bounce-netted ledger. * **caveman**: no delta material to this page (prompt-mechanism tool; hub pages 01/12 remain current); its reuse form is policy text, not code. * **Cross-cutting**: every tool's most expensive subsystem is the one jackin❯ doesn't need — per-agent installers, self-healing hooks, permission mediation, drift checkers (RTK 6.9k-line installer; lean-ctx 19 installers + healers; headroom `proxy_setup`/wrap plumbing; caveman hook delivery). Owning the runtime converts that entire class into config generation at capsule build. ## RTK — delta and reuse surface [#rtk--delta-and-reuse-surface] **Delta since 2026-06-18.** v0.43.0 (2026-06-28): `never_worse` output guard; guard-baseline authenticity fixes; OpenShift `oc` + Pulumi filters; git exit-code propagation fixes (#2494); grep engine no longer silently substituted. Develop (0.44-rc): PHP ecosystem (8 tools incl. phpunit state-machine parser), `uv run` transparent prefix, signal exit codes (128+n). 68,018★ (still 428:1 star:watcher — noise), **1,464 open issues**. README now claims 14 agent integrations; docs drift persists (develop manifests still say 0.42.4). RTK Cloud remains waitlist. **Source architecture (new to the hub).** 77.7k lines; no `lib.rs` — single binary crate, 3,535-line `main.rs` with a \~50-variant clap enum and manual 4-place wiring per native filter. Three extensibility tiers: declarative TOML filters (8-stage line pipeline: `strip_ansi → replace[] → match_output[] (with anti-swallow `unless`) → strip/keep_lines → truncate → head/tail → max_lines → on_empty`, project-local files content-hash trusted via `rtk trust`, inline `[[tests]]` + `rtk verify --require-all`); native Rust `StreamFilter`s (`feed_line/flush/on_exit` streaming, not buffer-everything); config exclude/transparent-prefix lists. Classifier: static 85-rule table (regex → rtk\_cmd + vendor savings\_pct) + shell lexer that splits `a && b` and passes through anything with backticks/`$( )`/redirects. Gain ledger: SQLite WAL (5s busy timeout for concurrent sessions), `len/4` estimator (`tracking.rs:1284`), vendor savings\_pct hardcoded into rules — both reasons its numbers stay T4. Hooks: 4 JSON dialect processors + **permission mediation** (mirrors the agent's allow/ask/deny; Default never auto-allows — issue #1155 invariant); `hooks/init.rs` alone is 6,894 lines of per-agent installer. Phone-home telemetry exists in official binaries (daily salted-hash ping, GDPR consent flow, inert in source builds) — a correction to any "no telemetry" reading. **Reuse verdict.** The crates.io name `rtk` is squatted (unrelated "Rust Type Kit"); upstream can't publish anyway (`license = "Apache 2.0"` is invalid SPDX). Third-party `rtk-registry` 1.0.0 (MIT relabel of the classifier) proves extraction demand but is a stale fork — use as a map, not a dependency. **Vendor**: TOML corpus + classifier + lexer + guard + tee + ParseResult (\~4–5k lines total, Apache-2.0-clean with NOTICE). **Port**: top-traffic native filters (cargo/git/gh/pytest/go-NDJSON/tsc/vitest) against the `StreamFilter` shape. **Skip**: the installer/hook/permission subsystem (obsoleted when the runtime owns PATH: shims are transparent, the agent's own permission prompt sees the original command), `len/4` accounting, hardcoded savings\_pct. **Bridge**: the binary's stable scriptable seams (`rtk rewrite` exit-code protocol 0/1/2/3, `rtk pipe --filter `, `rtk hook `) allow a shim-driven pilot before native filters land. New offline session-analysis suite (`rtk session`/`discover`/`learn`/`cc-economics` reading Claude Code JSONL) partially retires the hub's "no whole-session telemetry" line — adoption-%, not compression telemetry, but no longer nothing. ## headroom — delta and reuse surface [#headroom--delta-and-reuse-surface] **Delta since 2026-06-15.** Org move to `headroomlabs-ai` (GitHub redirect intact; badges/manifests partially unmigrated); "Headroom for teams" services offering (no SaaS API, no pricing page); license unchanged Apache-2.0; telemetry **opt-in** since 0.27.0; v0.28.0 adds pilot-hardening (inbound auth, audit log, air-gap switch). Model: `kompress-base` → **`kompress-v2-base`** (ModernBERT-base 149M + LoRA dual head, \~600 MB, Apache-2.0, still on the founder's personal HF account; no published ONNX). New: output-shaper with **holdout control group** + counterfactual savings CI; `learn --verbosity`; loop-detector weighting RTK re-fetch loops; per-project ledger; `--lossless` no-CCR proxy mode; xlsx compression; A/B accuracy eval framework. Aggregator posture: headroom now **bundles** RTK v0.42.4 and lean-ctx v3.4.7 and installs tokensave/Serena/codebase-memory as MCP servers — vendor-side confirmation of the hub's "layers, not competitors" thesis. **Implementation anatomy (corrects the hub's "Rust core" framing).** A 14-doc `REALIGNMENT/` plan (40 PRs, 9 phases) is \~⅔ landed: Rust crates `headroom-core` (tokenizers, CCR trait + SQLite/Redis/DashMap, 2,967-line `live_zone.rs`, full SmartCrusher port, log/search/diff/text compressors, 3-tier content detection: magika ONNX → unidiff parse-oracle → aho-corasick, BM25+fastembed relevance, `cache_control.rs`, `compression_policy.rs`), `headroom-py` (PyO3 — **the shipped hot path is Rust called from the Python proxy**), `headroom-proxy` (standalone axum + native SigV4/Vertex; built but not the shipped default), `headroom-parity` (Rust-vs-Python byte-parity gates). Phase H (retire Python) not executed. **ML retreat is explicit**: the synchronous kompress path caused a 30s-timeout → leaked-thread → executor-saturation cascade (#1171); mitigations are size gates, deadlines, opt-in background mode, and the deterministic **TextCrusher** (sentence scoring recency+BM25+salience; \~94% buried-answer retention at 30% size vs \~36% truncation; "sub-second where ModernBERT takes minutes"). In the Rust dispatcher `PlainText` is still a no-op (`TODO(PR-B4)`). **Now documented to spec grade** (the hub previously had these only as design sketches): live-zone exact algorithm — frozen floor = last explicit `cache_control` message index +1; ceiling = latest user message (latest assistant never touched); per-content-type 512 B thresholds; per-block token-validation gate; `RawValue` byte-range surgery (untouched bytes literally copied); `CompressionManifest` per block. CCR — blake3 first-24-hex keys, `<>` marker, 4-column SQLite WAL schema, lazy TTL purge, default TTL raised 5→30 min (expiry silently converts lossless→lossy). **Cache-economics gate** — `net_mutation_gain`/`break_even_reads` deciding whether a deep-block mutation is worth busting a warm prefix (1.25×/0.1× multipliers): the first vendor implementation of the hub's own break-even arithmetic. **Auth-mode policy** — Payg/OAuth/Subscription classified per request, gating lossy ratio and stealth behavior (with a fixed bug worth remembering: the old `sk-ant-oat-` prefix check misclassified every real subscription token as PAYG and aggressively compressed subscription traffic). **Correctness skeletons (their tracker, our test plan)**: #1307 `role:"tool"` outputs replaced with lossy *unrecoverable* summaries when CCR markers weren't emitted; #1077 infinite retrieve loop; #1006 markers emitted without the retrieve tool injected (silent data loss). Lesson class: reversibility guarantees are only as strong as marker⇄tool-injection⇄TTL coupling. **Reuse verdict.** No crates.io publication; 0.1.0 parity-constrained crates make a git dependency weak. **Reimplement primitives 1–4 (live-zone, CCR with wire-compatible markers, typed router, auth-mode + economics gate) from the now-spec-grade docs, using headroom's own crate choices** (tiktoken-rs, magika, unidiff+panic-guard, aho-corasick, blake3, rusqlite-class store, rayon); skip the ML path and the whole Python stack; keep operator-side headroom-MCP as an A/B arm. ## lean-ctx — delta and reuse surface [#lean-ctx--delta-and-reuse-surface] **Delta since 2026-06-20 (v3.8.11 → 3.8.18 + HEAD).** Highlights: addon ecosystem (registry, Ed25519 signing, optional OS sandbox); QUBO context-selection spike **rejected** (greedy knapsack parity — stays default); embeddings migrated to ort/ONNX **runtime-dylib** (not linked); `json_crush` — an explicit deterministic **port of headroom's SmartCrusher**; Anthropic cache-breakpoint injection + cache-aligner telemetry; CCR-coupled auto-degrade (expand rate dials compression down); persistent read-stub index + conversation-scoped stubs + `task:{id}` subagent scoping; write-time memory dedup-admission; stable cross-turn symbol handles; +5 signature languages → **26 grammars**; Context Time Machine (BLAKE3 content-addressed, ed25519-signed layer snapshots); lossless memory (eviction archive-backed); `ctx_outline` made honest after the ast-grep author publicly called its claims "fishy" (per-file `backend: tree-sitter|regex` labels); **reversibility as first-class recovery** (#625: `RECOVER` rule, recovery footer on every lossy read, default high-compression tee — root cause of "too compressed" complaints was undiscoverable escape hatches); rmcp 2.0; hook-aware honest Cursor guidance (retreat from "NEVER use native tools"; rule payload −55%); PostToolUse read-dedup for Claude Code (`[unchanged]` stub via `updatedToolOutput`); hybrid multi-repo search; pgvector backend via `psql` (zero new crates); PDF/Obsidian ingestion. Self-footprint is now a **CI-gated headline metric** (`doctor overhead --gate`, \~2.1K tokens/session). **Corrections to hub pages (source-verified).** (1) "77 MCP tools" is marketing noise — registry \~81–83, but the **advertised default is 5 tools + a universal `ctx_call` invoker**; everything else is callable-but-unadvertised (zero schema tokens), with pinned profiles and dynamic categories. (2) "No independent benchmark" — obsolete: **tokbench R1** exists and found the default config *taxing* (3K-token prefix re-billed per turn; shell routing 0% engaged); upstream now ships a "faithful arm" minimal profile + preflight gate. (3) "Several SQLite stores" overstated — core persistence is JSON + postcard/zstd; SQLite only for graph/event-bus/FTS. (4) The 64.7 MB binary figure is stale post-#497 (ONNX no longer linked; upstream claims \~17 MB class) — re-measure before quoting. (5) Tokenizer: multi-tokenizer (o200k primary, cl100k for Claude ±3% claimed, Gemini 1.1×), still GPT-family — caveat stands, sharpened. **Reuse verdict.** Workspace is **one monolith crate** (307 files under `core/` alone) + a git-only SDK facade that pins the heavy features — neither is a healthy dependency. **Port kernels with attribution**: read-stub cache + persistent stub index + scoping (highest ROI, smallest surface); deterministic crushers + 56 shell pattern modules (as references beside RTK's corpus); `signatures_ts` declarative tree-sitter queries with per-role grammar linking; CFT Φ + greedy-knapsack compiler (pure data model, signals injected); bounce-netted signed ledger semantics. **Adopt doctrines**: recovery-first; lazy tool surface + CI-gated self-footprint; measurement-before-mutation ("mutation opt-in, measurement default-on" proxy posture); net-of-injection accounting; corpus-signature freshness over TTLs. **Reject**: daemon/dashboard/proxy/19 installers + self-healing hooks + drift checkers (the tax of instrumenting a runtime you don't own), cognition sprawl, in-process embeddings. Optional: the binary as a role addon in faithful-arm profile. ## The consolidated "runtime owner's dividend" [#the-consolidated-runtime-owners-dividend] Placing the four tools' subsystems into two buckets makes the engine case quantitative rather than rhetorical: | Bucket | RTK | headroom | lean-ctx | caveman | | ----------------------------------------------------------------- | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------- | | **Mechanism** (compresses/retrieves/measures) | filters, classifier, guard, tee | live-zone, CCR, router, economics gate | stubs, crushers, signatures, CFT, ledger | register text | | **Delivery tax** (exists only because they don't own the runtime) | 6.9k-line installer, 4 hook dialects, permission mediation, trust/integrity checks | proxy + wrap plumbing, client-config mutation, stealth auth sniffing | 19 installers, self-healing hooks, drift detection, uninstall strippers, rules re-sync | hook delivery, per-turn re-injection, stats hooks | Every row-2 subsystem disappears under a runtime that composes capsule config at launch — which is the [engine spec](/reference/research/jackin-context-engine/)'s founding argument, and why "embed the mechanisms, skip the delivery" is not a rewrite-everything fantasy: the mechanisms are the *small* halves of these codebases. ## Ledger [#ledger] | Claim | Basis | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | RTK v0.43.0 features, filter/rule counts, ledger schema, telemetry, installer size | clone `develop` @ `b52e02b` 2026-07-01, checked 2026-07-03; installed binary 0.43.0 | | rtk crates.io squat; invalid SPDX; rtk-registry/mcp-rtk third-party crates | crates.io API, 2026-07-03 | | headroom org move, versions, REALIGNMENT status, live-zone/CCR internals, ML retreat, bug classes | clone @ `c75ebde` 2026-07-02; PyPI/npm 0.28.0; HF kompress-v2 pages (2026-07-03) | | lean-ctx 3.8.12–3.8.18 deltas, tool-surface mechanics, tokbench R1, kernel map | clone @ `a80350da` 2026-07-03; CHANGELOG; crates.io `lean-ctx` 3.8.18 | | sem/mq findings referenced | [hub 25](/reference/research/token-optimization-tools/25-sem-and-mq-structural-and-markdown-retrieval/) |