13 — Reuse surfaces and deltas (2026-07-03 source sweep)

This page records the 2026-07-03 re-sweep of the hub's four tools — fresh clones, registry checks, release deltas since the 06-15/06-18/06-20 snapshots — through a new lens the hub had not applied: the Rust reuse surface. The motivating question comes from the jackin❯ side: instead of running these tools in a container, which of their mechanisms can be embedded natively (crate dependency, vendored corpus, or ported kernel) into one engine? The resulting product design lives in the jackin❯ Context Engine spec; two tools new to the research (sem, mq) are covered in hub 25 — sem and mq. Everything below is source-verified with the same T1–T4 discipline; corrections to earlier hub pages are consolidated at the end.

TL;DR

• RTK (v0.43.0; develop @ b52e02b): still not a library, but its highest-value assets are cleanly vendorable — 64 declarative TOML filters with inline tests, an 85-rule command classifier + shell lexer with an unattestable-construct guard, and two new safety contracts worth adopting anywhere: the v0.43 never_worse guard (filtered ≤ raw or emit raw) and three-tier ParseResult degradation (Full/Degraded/Passthrough, "never return false data silently"). The hub's "12 strategies" was a taxonomy; the real dispatch surface is 42 native command modules + 64 TOML filters + 85 rewrite rules.
• headroom (v0.28.0; moved chopratejas → headroomlabs-ai, Apache-2.0 unchanged, telemetry now opt-in): a phased Python→Rust rewrite is ~⅔ executed — the typed-compressor hot path is genuinely Rust today (PyO3 inside the Python proxy) — but nothing is on crates.io and the crates are parity-constrained 0.1.0. The ML stage is being demoted by upstream itself (timeout cascades; deterministic TextCrusher replacement), vindicating the hub's caution. The live-zone algorithm, CCR schema (blake3/24-hex, <<ccr:HASH>>), cache-economics gate (net_mutation_gain, 1.25×/0.1×), and auth-mode policy are now documented to reimplementation grade.
• lean-ctx (v3.8.18): now published on crates.io — but with explicitly no internal-API stability, and the curated SDK facade is publish = false git-only, pinning tree-sitter×26 + ort + http into any consumer. The hub's "no independent benchmark" is obsolete, unflatteringly: the pre-registered tokbench R1 ran lean-ctx at defaults and found a ~3K-token injected-prefix tax re-billed every turn and 102 native bash / 0 ctx_shell (routing never engaged). Upstream answered with a "faithful arm" minimal profile. Lesson elevated to a spec invariant: savings must be reported net-of-injection. Portable kernels: read-stub cache, deterministic crushers, signatures_ts, CFT knapsack compiler, bounce-netted ledger.
• caveman: no delta material to this page (prompt-mechanism tool; hub pages 01/12 remain current); its reuse form is policy text, not code.
• Cross-cutting: every tool's most expensive subsystem is the one jackin❯ doesn't need — per-agent installers, self-healing hooks, permission mediation, drift checkers (RTK 6.9k-line installer; lean-ctx 19 installers + healers; headroom proxy_setup/wrap plumbing; caveman hook delivery). Owning the runtime converts that entire class into config generation at capsule build.

RTK — delta and reuse surface

Delta since 2026-06-18. v0.43.0 (2026-06-28): never_worse output guard; guard-baseline authenticity fixes; OpenShift oc + Pulumi filters; git exit-code propagation fixes (#2494); grep engine no longer silently substituted. Develop (0.44-rc): PHP ecosystem (8 tools incl. phpunit state-machine parser), uv run transparent prefix, signal exit codes (128+n). 68,018★ (still 428:1 star:watcher — noise), 1,464 open issues. README now claims 14 agent integrations; docs drift persists (develop manifests still say 0.42.4). RTK Cloud remains waitlist.

Source architecture (new to the hub). 77.7k lines; no lib.rs — single binary crate, 3,535-line main.rs with a ~50-variant clap enum and manual 4-place wiring per native filter. Three extensibility tiers: declarative TOML filters (8-stage line pipeline: strip_ansi → replace[] → match_output[] (with anti-swallow unless) → strip/keep_lines → truncate → head/tail → max_lines → on_empty, project-local files content-hash trusted via rtk trust, inline [[tests]] + rtk verify --require-all); native Rust StreamFilters (feed_line/flush/on_exit streaming, not buffer-everything); config exclude/transparent-prefix lists. Classifier: static 85-rule table (regex → rtk_cmd + vendor savings_pct) + shell lexer that splits a && b and passes through anything with backticks/$( )/redirects. Gain ledger: SQLite WAL (5s busy timeout for concurrent sessions), len/4 estimator (tracking.rs:1284), vendor savings_pct hardcoded into rules — both reasons its numbers stay T4. Hooks: 4 JSON dialect processors + permission mediation (mirrors the agent's allow/ask/deny; Default never auto-allows — issue #1155 invariant); hooks/init.rs alone is 6,894 lines of per-agent installer. Phone-home telemetry exists in official binaries (daily salted-hash ping, GDPR consent flow, inert in source builds) — a correction to any "no telemetry" reading.

Reuse verdict. The crates.io name rtk is squatted (unrelated "Rust Type Kit"); upstream can't publish anyway (license = "Apache 2.0" is invalid SPDX). Third-party rtk-registry 1.0.0 (MIT relabel of the classifier) proves extraction demand but is a stale fork — use as a map, not a dependency. Vendor: TOML corpus + classifier + lexer + guard + tee + ParseResult (~4–5k lines total, Apache-2.0-clean with NOTICE). Port: top-traffic native filters (cargo/git/gh/pytest/go-NDJSON/tsc/vitest) against the StreamFilter shape. Skip: the installer/hook/permission subsystem (obsoleted when the runtime owns PATH: shims are transparent, the agent's own permission prompt sees the original command), len/4 accounting, hardcoded savings_pct. Bridge: the binary's stable scriptable seams (rtk rewrite exit-code protocol 0/1/2/3, rtk pipe --filter <name>, rtk hook <agent>) allow a shim-driven pilot before native filters land. New offline session-analysis suite (rtk session/discover/learn/cc-economics reading Claude Code JSONL) partially retires the hub's "no whole-session telemetry" line — adoption-%, not compression telemetry, but no longer nothing.

headroom — delta and reuse surface

Delta since 2026-06-15. Org move to headroomlabs-ai (GitHub redirect intact; badges/manifests partially unmigrated); "Headroom for teams" services offering (no SaaS API, no pricing page); license unchanged Apache-2.0; telemetry opt-in since 0.27.0; v0.28.0 adds pilot-hardening (inbound auth, audit log, air-gap switch). Model: kompress-base → kompress-v2-base (ModernBERT-base 149M + LoRA dual head, ~600 MB, Apache-2.0, still on the founder's personal HF account; no published ONNX). New: output-shaper with holdout control group + counterfactual savings CI; learn --verbosity; loop-detector weighting RTK re-fetch loops; per-project ledger; --lossless no-CCR proxy mode; xlsx compression; A/B accuracy eval framework. Aggregator posture: headroom now bundles RTK v0.42.4 and lean-ctx v3.4.7 and installs tokensave/Serena/codebase-memory as MCP servers — vendor-side confirmation of the hub's "layers, not competitors" thesis.

Implementation anatomy (corrects the hub's "Rust core" framing). A 14-doc REALIGNMENT/ plan (40 PRs, 9 phases) is ~⅔ landed: Rust crates headroom-core (tokenizers, CCR trait + SQLite/Redis/DashMap, 2,967-line live_zone.rs, full SmartCrusher port, log/search/diff/text compressors, 3-tier content detection: magika ONNX → unidiff parse-oracle → aho-corasick, BM25+fastembed relevance, cache_control.rs, compression_policy.rs), headroom-py (PyO3 — the shipped hot path is Rust called from the Python proxy), headroom-proxy (standalone axum + native SigV4/Vertex; built but not the shipped default), headroom-parity (Rust-vs-Python byte-parity gates). Phase H (retire Python) not executed. ML retreat is explicit: the synchronous kompress path caused a 30s-timeout → leaked-thread → executor-saturation cascade (#1171); mitigations are size gates, deadlines, opt-in background mode, and the deterministic TextCrusher (sentence scoring recency+BM25+salience; ~94% buried-answer retention at 30% size vs ~36% truncation; "sub-second where ModernBERT takes minutes"). In the Rust dispatcher PlainText is still a no-op (TODO(PR-B4)).

Now documented to spec grade (the hub previously had these only as design sketches): live-zone exact algorithm — frozen floor = last explicit cache_control message index +1; ceiling = latest user message (latest assistant never touched); per-content-type 512 B thresholds; per-block token-validation gate; RawValue byte-range surgery (untouched bytes literally copied); CompressionManifest per block. CCR — blake3 first-24-hex keys, <<ccr:HASH>> marker, 4-column SQLite WAL schema, lazy TTL purge, default TTL raised 5→30 min (expiry silently converts lossless→lossy). Cache-economics gate — net_mutation_gain/break_even_reads deciding whether a deep-block mutation is worth busting a warm prefix (1.25×/0.1× multipliers): the first vendor implementation of the hub's own break-even arithmetic. Auth-mode policy — Payg/OAuth/Subscription classified per request, gating lossy ratio and stealth behavior (with a fixed bug worth remembering: the old sk-ant-oat- prefix check misclassified every real subscription token as PAYG and aggressively compressed subscription traffic).

Correctness skeletons (their tracker, our test plan): #1307 role:"tool" outputs replaced with lossy unrecoverable summaries when CCR markers weren't emitted; #1077 infinite retrieve loop; #1006 markers emitted without the retrieve tool injected (silent data loss). Lesson class: reversibility guarantees are only as strong as marker⇄tool-injection⇄TTL coupling.

Reuse verdict. No crates.io publication; 0.1.0 parity-constrained crates make a git dependency weak. Reimplement primitives 1–4 (live-zone, CCR with wire-compatible markers, typed router, auth-mode + economics gate) from the now-spec-grade docs, using headroom's own crate choices (tiktoken-rs, magika, unidiff+panic-guard, aho-corasick, blake3, rusqlite-class store, rayon); skip the ML path and the whole Python stack; keep operator-side headroom-MCP as an A/B arm.

lean-ctx — delta and reuse surface

Delta since 2026-06-20 (v3.8.11 → 3.8.18 + HEAD). Highlights: addon ecosystem (registry, Ed25519 signing, optional OS sandbox); QUBO context-selection spike rejected (greedy knapsack parity — stays default); embeddings migrated to ort/ONNX runtime-dylib (not linked); json_crush — an explicit deterministic port of headroom's SmartCrusher; Anthropic cache-breakpoint injection + cache-aligner telemetry; CCR-coupled auto-degrade (expand rate dials compression down); persistent read-stub index + conversation-scoped stubs + task:{id} subagent scoping; write-time memory dedup-admission; stable cross-turn symbol handles; +5 signature languages → 26 grammars; Context Time Machine (BLAKE3 content-addressed, ed25519-signed layer snapshots); lossless memory (eviction archive-backed); ctx_outline made honest after the ast-grep author publicly called its claims "fishy" (per-file backend: tree-sitter|regex labels); reversibility as first-class recovery (#625: RECOVER rule, recovery footer on every lossy read, default high-compression tee — root cause of "too compressed" complaints was undiscoverable escape hatches); rmcp 2.0; hook-aware honest Cursor guidance (retreat from "NEVER use native tools"; rule payload −55%); PostToolUse read-dedup for Claude Code ([unchanged] stub via updatedToolOutput); hybrid multi-repo search; pgvector backend via psql (zero new crates); PDF/Obsidian ingestion. Self-footprint is now a CI-gated headline metric (doctor overhead --gate, ~2.1K tokens/session).

Corrections to hub pages (source-verified). (1) "77 MCP tools" is marketing noise — registry ~81–83, but the advertised default is 5 tools + a universal ctx_call invoker; everything else is callable-but-unadvertised (zero schema tokens), with pinned profiles and dynamic categories. (2) "No independent benchmark" — obsolete: tokbench R1 exists and found the default config taxing (3K-token prefix re-billed per turn; shell routing 0% engaged); upstream now ships a "faithful arm" minimal profile + preflight gate. (3) "Several SQLite stores" overstated — core persistence is JSON + postcard/zstd; SQLite only for graph/event-bus/FTS. (4) The 64.7 MB binary figure is stale post-#497 (ONNX no longer linked; upstream claims ~17 MB class) — re-measure before quoting. (5) Tokenizer: multi-tokenizer (o200k primary, cl100k for Claude ±3% claimed, Gemini 1.1×), still GPT-family — caveat stands, sharpened.

Reuse verdict. Workspace is one monolith crate (307 files under core/ alone) + a git-only SDK facade that pins the heavy features — neither is a healthy dependency. Port kernels with attribution: read-stub cache + persistent stub index + scoping (highest ROI, smallest surface); deterministic crushers + 56 shell pattern modules (as references beside RTK's corpus); signatures_ts declarative tree-sitter queries with per-role grammar linking; CFT Φ + greedy-knapsack compiler (pure data model, signals injected); bounce-netted signed ledger semantics. Adopt doctrines: recovery-first; lazy tool surface + CI-gated self-footprint; measurement-before-mutation ("mutation opt-in, measurement default-on" proxy posture); net-of-injection accounting; corpus-signature freshness over TTLs. Reject: daemon/dashboard/proxy/19 installers + self-healing hooks + drift checkers (the tax of instrumenting a runtime you don't own), cognition sprawl, in-process embeddings. Optional: the binary as a role addon in faithful-arm profile.

The consolidated "runtime owner's dividend"

Placing the four tools' subsystems into two buckets makes the engine case quantitative rather than rhetorical:

Every row-2 subsystem disappears under a runtime that composes capsule config at launch — which is the engine spec's founding argument, and why "embed the mechanisms, skip the delivery" is not a rewrite-everything fantasy: the mechanisms are the small halves of these codebases.

Ledger