# 09 — Reuse decision matrix (https://jackin.tailrocks.com/reference/research/jackin-context-engine/09-reuse-decision-matrix/) # 09 — Reuse decision matrix [#09--reuse-decision-matrix] Per-tool build-vs-buy decisions from the 2026-07-03 source sweep ([full per-tool findings](/reference/research/token-optimization-tools/13-reuse-surfaces-and-deltas/)); version pins and upstream-status facts follow the [latest-versions refresh](/reference/research/token-optimization-tools/18-latest-versions-refresh/). Workspace constraint: `deny.toml` allows crates.io sources with an Apache-2.0 + MIT license allowlist — every dependency below clears it. All pins exact (`=x.y.z`); pre-1.0 upstreams get reviewed bumps via Renovate, never `^`. ## Decision table [#decision-table] | Tool | Language / license | Decision | What, exactly | | ---------------- | ----------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **fff** | Rust, MIT | **DEPEND** | `fff-search = "=0.9.6"` (default-features off → pure-Rust walker; skip `zlob`/Zig), `fff-query-parser = "=0.9.6"`. One `SharedFilePicker` per workspace mount; frecency LMDB under `/jackin/state/context/index/`. Production-proven: nushell pins `=0.9.6`. Stable pin unchanged, but `0.9.7-nightly.*` prereleases are publishing daily and PR #645 rewrites exactly the per-platform file walker + gitignore matcher — the default-features-off pure-Rust-walker guidance needs a deliberate re-check before any 0.9.7 bump. **PORT** the bin-only agent-formatting layer (\~800 LOC: frecency words, def-first context, import de-noise, cursor pagination) — upstream PR candidate. Caveats: pre-1.0 churn, author-controlled forks (`neo_frizbee`, notify-debouncer) in the tree, vendored libgit2 + LMDB native deps, `notify` at RC — cargo-vet the subtree. Note: the crates.io name `fff` is an unrelated squatter; the crate is **`fff-search`**. | | **sem** | Rust, MIT/Apache-2.0 | **DEPEND (core) + REIMPLEMENT (ledger)** | `sem-core = "=0.18.0"` with per-role `lang-*` grammar features: entity extraction (38 formats), two-pass entity graph, structural-hash diff (`compute_semantic_diff` — rename/move/cosmetic detection), budgeted context packer (`build_context_result_bounded`), orient ranking. API note carried from the 0.16→0.18 jump: 0.16.0 added a public `omitted` field to `ContextResult`, breaking struct-literal construction — code against constructors, not literals; 0.17/0.18 add unique-method-name call edges (Python/Ruby), `Parent::child` qualifiers, and `sem orient --pack` turn-zero packed briefings (relevant to the read-stub/prefetch layer). Externally consumed already (weave, inspect). The **attention ledger / delta-fill** machinery lives in bin-first `sem-mcp` — reimplement natively in the capsule supervisor (\~small, spec documented upstream), generalized to all tool traffic. **SKIP**: sem-cloud (fly.dev egress), telemetry (default-on, opt out regardless), self-update, `sem setup` git rewiring, the Read/Grep-denying guard hook. Watch: cli/mcp crates.io publishing still wedged — sem-cli/sem-mcp frozen at 0.14.1 while sem-core ships 0.18.0, `sem-cloud-client` still unpublished; 0.x breaking minors (two in the 19 hours after the 07-03 sweep). | | **mq** | Rust, MIT | **DEPEND** | `mq-lang = { version = "=0.6.4", features = ["sync"] }` (Send/Sync engine; leave `http-import` and the new default-off `http-import-ureq` both off — kills remote-module supply chain), `mq-markdown = { version = "=0.6.4", features = ["json"] }` (19-crate minimal tree, nushell-proven) for chunking-only paths. Compile canned programs once (`Engine::compile`/`eval_compiled`): outline, section, skeleton, code-blocks. Ship the `mq` binary + its processing-markdown skill in capsule images as the agent-facing escape hatch (0.6.4 adds CLI `--count` and `--skip`/`--limit` result pagination — directly useful for budgeted escape-hatch queries). Caveats: bus factor 1, weekly 0.x releases, \~291-crate tree for full `mq-lang` (no format gating upstream — a features PR candidate); `mq-db` is a 0.1.x prototype worth watching for index design, not depending on. | | **RTK** | Rust, Apache-2.0 | **VENDOR (corpus) + PORT (kernels)** | Not a library (no lib.rs; crates.io name taken by an unrelated project; invalid SPDX string blocks upstream publish). Vendor with attribution: the **63 built-in TOML filters** (with inline `[[tests]]` — the single highest-value asset), the **86-rule classifier + shell lexer** (incl. quote-aware unattestable-construct guard and `transparent_prefixes`), the 8-stage TOML pipeline with `unless` anti-swallow guards, `never_worse` guard, three-tier `ParseResult` (+ `[N more lines]` anti-confusion end-marker contract, per-filter panic isolation, `reduced()` cap guards, content-sniffing pipe detection, `extract_json_object`, TOCTOU-safe trust-before-load, integrity sidecars). Port the high-traffic native filters (cargo/git/gh/pytest/go-NDJSON/tsc/vitest) against the `StreamFilter` trait shape (77 CLI commands / 62 modules upstream — port by traffic, not parity). **SKIP**: the 6.9k-line installer class (obsoleted by shims), permission mediation, `chars/4` ledger, hardcoded vendor savings\_pct, telemetry (consent-gated upstream, but its `low_savings_commands` field can leak command arguments — the engine sends nothing). Bridge option: bake the rtk binary into capsule images and drive `rtk pipe --filter` / `rtk rewrite` from shims until native filters land — proven seams, trivially removable. | | **headroom** | Python + Rust, Apache-2.0 | **REIMPLEMENT to documented spec** | No published crates (upstream at v0.30.0 under `headroomlabs-ai`; workspace crates 0.1.0, path-deps, parity-with-Python constraints baked in; `headroom` crate name squatted by an audio tool). Reimplement natively using their own documented designs + crate choices: **live-zone dispatcher** (frozen-floor from `cache_control`, latest-user-message ceiling, per-type thresholds, token-validation gate, byte-range surgery, per-block manifest), **CCR store** (exact schema + `<>` blake3 wire format for interop with the operator's running headroom MCP), **typed router** (magika ONNX tier optional → unidiff parse-oracle with panic guard → aho-corasick keywords), **cache-economics gate** (`net_mutation_gain`, 1.25×/0.1× multipliers), **auth-mode policy**, **holdout counterfactual measurement** (conversation-stable blake3 cohorts + online-variance CI), the **lossless-compaction tier with runtime round-trip self-check** (`--lossless` mode, shipped upstream in v0.29.0), **TextCrusher** (deterministic extractive prose scoring — already Rust in `headroom-core`), the corrected `net_mutation_gain` (penalty over `S+ΔT`), and the TOIN observe-only/offline-publish learning architecture with subscription write-quarantine. Their bug classes (#1006 marker-without-tool, #1077 retrieve loop, #1307 lossy tool-output summaries) become the engine's test plan. **SKIP**: kompress ML models (gated upstream, still default fallback — the engine has no ML tier at all), the Python proxy/memory/graph stack (memory schema: only 2/6 tables agent-scoped, no migrations — counterexample), the `max(cli_filtering, rtk)` savings collapse (unattributable double-counting), unpinned git-main MCP installs, bundling of RTK/lean-ctx binaries. Evidence note: headroom's "independent 47.5%" was a conflation with its own self-benchmark and its 4.8%-median fleet telemetry is historical (beacon removed) — its evidence tier is now self-report-plus-academic-backing, same class as the others. Keep operator-side headroom-MCP as an A/B comparison arm. | | **lean-ctx** | Rust, Apache-2.0 | **PORT kernels; optional sidecar binary** | Published on crates.io (3.8.18; v3.9.0 cut in-repo 2026-07-04 — AI Gateway team mode, grammar-addon dylibs, agents-only hook default — but unreleased and unpublished) but explicitly no internal-API stability, and the SDK facade is git-only (`publish = false`) pinning tree-sitter×26 + ort + http into consumers — the monolith in library clothing. Port with attribution: **read-stub cache** (`core/cache.rs` + `read_stub_index` + conversation/`task:` scoping — highest ROI, smallest dep surface), **deterministic crushers** (`json_crush`, tabular/yaml, the 77 shell-pattern modules — count corrected round-2 — as filter references), **`signatures_ts` outline engine** (declarative per-language tree-sitter queries; link per-role grammar subsets), **CFT Φ-scoring + greedy-knapsack context compiler** (pure data model; QUBO spike confirmed greedy suffices), savings-ledger bounce netting (5-tick window, 0.30/≥3 force-full, 10-tick post-edit exclusion). Adopt doctrines: recovery footers on every lossy read, lazy tool surface + `doctor overhead --gate` (their minimal-arm prefix budget is CI-pinned at 1,835 tokens — ours must beat it), measurement-before-mutation, net-of-injection accounting, corpus-signature freshness, **deterministic-argmax learning for cache stability (#498)**, the mode-default table (docs never compressed; no lossy override), `anchored` per-line-hash read mode, the Kolmogorov gzip gate, footprint self-ablation (#959). Round-2 count corrections: **13** read modes; release binary measured **75.9 MiB**. **SKIP**: daemon/dashboard/proxy/19 installers/cognition sprawl/in-process ort embeddings/SPLADE (watch)/A2A HMAC transport (upstream's endpoint never verifies)/scent-field stigmergy (novel zero-token multi-agent coordination — record, revisit with the fleet story). Optional: ship the `lean-ctx` binary as a role addon preconfigured with the faithful-arm profile (`tool_profile=minimal`, `rules_injection=off`) for users wanting the full engine — its defaults demonstrably tax harness workloads. | | **codedb** | Zig, BSD-3-Clause | **DESIGN DONOR (reimplement)** | No Rust linkage path worth the FFI cost (Zig 0.16 toolchain, mutex-guarded stateful engine, alpha snapshot format at v0.2.5827, bus factor 1). Take the designs: query-verb grammar + output-shaping idioms (skeleton `{ … N lines }` elision, escalation footers, `unchanged:HASH`, `paths_only`, pagination tails, one-line omission markers), value-ordered budget packing (`handleContext` two-step pack), additive-never-filter ranking stack with per-signal kill switches (eval-tuned on SWE-bench-Lite sweeps), warm-daemon + snapshot economics, agent-registry file-lock primitive. Rust equivalents per capability: tree-sitter (outlines — better than codedb's line heuristics), tantivy or roaring-bitmap trigrams + `bm25` (search), `fst`/`nucleo` (symbol/fuzzy), `ignore`+`globset` (walker), `notify` (watcher — better than 2s polling), `xxhash-rust`/`blake3` (change detection), `memmap2`+`rkyv` (snapshots), `petgraph` (call graph + PageRank), `gix` (co-change, no shell-out), `rmcp` (optional MCP facade). Round-2 additions: **config-fingerprinted cache keys**, **ConvergenceGovernor** anti-loop ring, handler-path-exact **warmup replay**, the reader.md hardening set (traversal rejection, entry caps, LOC ceiling, golden hash-parity tests, length-gated prepend), the **failed-experiments ledger** practice, two-stage mtime→hash change detection, per-response token cost as a benchmark axis, and the OpenAI-strict `oneOf` lesson (keep composition server-side). Anti-lessons from its own honest benchmarks: plain FTS5 beat it end-to-end once (efficiency > structural richness); don't hold full contents in RAM; gate every smart feature on a measured eval. Its install surface is the trust-boundary anti-pattern spec: undocumented global command-blocking hook, self-granted permissions, competitor hook-fronting, **default-on telemetry uploaded to the vendor on session close** — the engine does none of these by construction. | | **caveman** | JS plugin (prompts), MIT | **NATIVE PROMPT PACK** | No code to reuse — the mechanism is instruction text. Adopt the register design (graduated levels, verbatim guards, auto-clarity carve-outs, subagent report registers) as role-generated policy with the corrected numbers (58.5% ultra on visible prose; \~50% over terse control; format-enforcement does the work, politeness asks do nothing); upstream v1.9.1 has itself retired the \~75% headline for a flat "65% average" — the engine's policy keeps the measured numbers regardless. **SKIP**: hooks, \~940-token skill listing, per-turn re-injection, wenyan, `/caveman-stats` (superseded by the ledger), and `/caveman-commit`-style compression of commit/PR text (see caveman-code's exemption — the correct policy). | | **caveman-code** | TS coding agent (pi-code fork), MIT | **CONSTANTS + HARNESS DONOR** | Not a dependency (it's a competing agent). Take: the register **exception list** (code, commit messages, PR descriptions, security/destructive text — exempt at every level); the per-tool line-budget constants (bash 80 head50/tail30, read 300/200/100, grep 120/80/40, fallback 150) as the L2 defense-in-depth tier; read-dedup-by-fingerprint stubs (convergent with the sem/lean-ctx attention-ledger design the engine already adopts); the subagent output contract ("file:line over prose, no emojis, terse"); and the **honest-bench harness template** (real child processes, per-task `verify.sh`, fresh-token accounting, config-fingerprint column, committed raw artifacts) for the engine's cross-tool A/Bs. Its 1.93×-vs-Codex number is real but full-stack at non-default aggressiveness with −1/−2 task quality — cite with caveats only. | | **cavemem** | TS MCP memory (unmaintained), MIT | **REQUIREMENTS DONOR (do not depend)** | Refuted as shipped: 33–36% measured vs 75% marketed, savings refunded by default re-expansion at read time, FTS vocabulary mismatch, global cross-project retrieval, uncapped unmeasured injection, dead privacy config; unmaintained since 2026-05. Take exactly one artifact — the **preserved-segment tokenizer** (\~100 lines, byte-exact technical-token pass-through, ported as the engine's prose-transform safety contract) — and its four defects as L5 requirements (index canonical form; embed canonical; project-scoped retrieval; budgeted+measured injection). Its \~300-token 4-tool MCP surface is a good leanness example. | | **pxpipe** | TypeScript, MIT | **METHOD DONOR (no code reuse)** | Nothing DEPENDable (npm-only `pxpipe-proxy`, pre-1.0, \~6.5 weeks old, bus factor 1). Take as L0 prior art: the **counterfactual savings-accounting design** — per-request `count_tokens` probe on the original uncompressed body, baseline priced under the **same observed cache state** as the billed request, negative rows never floored — the anti-K1 gold standard in the surveyed field and exactly the shape the engine's ledger must match. Also take its cache-alignment invariants as design rules for any prompt-mutating feature: never add `cache_control` markers (relocate the caller's existing one to the end of rewritten stable content), freeze rewritten prefixes at first render, quantize any moving old/new boundary, price mode-flips with symmetric cache-burn terms. Record **optical/text-to-image context re-encoding** as a harness-gated frontier experiment only, never a default: lossy with a silent-confabulation failure shape (byte-exact strings must never live only in pixels), economics and legibility hostage to provider vision-pricing/resample behavior (its own measurements already flip negative across Opus 4.8 / GPT 5.5), and the whole-prompt-rewrite proxy form violates the engine's INV-8/INV-10-class no-proxy constraints. **SKIP**: the proxy itself, image re-encoding of anything byte-exact. Full teardown: [17 — pxpipe: design teardown](/reference/research/token-optimization-tools/17-pxpipe-design/). | ## Consolidated dependency delta (workspace additions) [#consolidated-dependency-delta-workspace-additions] ```toml # crates/jackin-context/Cargo.toml (new) fff-search = { version = "=0.9.6", default-features = false } fff-query-parser = { version = "=0.9.6", default-features = false } sem-core = { version = "=0.18.0", default-features = false, features = ["git", "parallel"] } # + per-role lang-* features; ContextResult gained a public `omitted` field in 0.16.0 — construct via constructors, not struct literals mq-lang = { version = "=0.6.4", features = ["sync"] } # http-import AND http-import-ureq stay off mq-markdown = { version = "=0.6.4", features = ["json"] } tree-sitter = "0.26" # + per-role grammar crates (MIT — deny.toml clean) tiktoken-rs = "0.12" # local estimation only; billing-grade counts via count_tokens API blake3 = "1" aho-corasick = "1" petgraph = "0.8" # optional: call-graph feature gix = "0.73" # co-change signals, no git shell-out notify = "8" # watcher (stable release, not the RC fff pins internally) # storage: turso (already in workspace via jackin-usage) ``` Vendored (with NOTICE attribution, Apache-2.0→Apache-2.0): RTK `filters/*.toml` corpus + classifier rule table; headroom live-zone/CCR constants and thresholds transcribed from spec; lean-ctx pattern-module references. Explicitly not added: tantivy (revisit only if BM25-over-hashmap postings proves insufficient), ort/embedding runtimes (non-goal), any Python. ## Upstream contributions worth filing [#upstream-contributions-worth-filing] 1. fff: publish the agent-formatting layer as a lib crate (or feature of `fff-search`); declare `rust-version`; CHANGELOG. 2. sem: split `sem-mcp`'s cache/ledger into a published library crate; unwedge cli/mcp publishing (`sem-cloud-client`). 3. mq: cargo features to gate data-format builtins (shrinks the 291-crate tree); publish `mq-lint`. 4. RTK: fix the invalid `license = "Apache 2.0"` SPDX string; publish `rtk-filters` (TOML corpus + engine) as a crate — the community fork ecosystem shows the demand. 5. headroom: publish `headroom-core` to crates.io once the parity constraints drop.