Researchjackin❯ Context Engine (spec)
09 — Reuse decision matrix
09 — Reuse decision matrix
Per-tool build-vs-buy decisions from the 2026-07-03 source sweep (full per-tool findings); version pins and upstream-status facts follow the latest-versions refresh. Workspace constraint: deny.toml allows crates.io sources with an Apache-2.0 + MIT license allowlist — every dependency below clears it. All pins exact (=x.y.z); pre-1.0 upstreams get reviewed bumps via Renovate, never ^.
Decision table
| Tool | Language / license | Decision | What, exactly |
|---|---|---|---|
| fff | Rust, MIT | DEPEND | fff-search = "=0.9.6" (default-features off → pure-Rust walker; skip zlob/Zig), fff-query-parser = "=0.9.6". One SharedFilePicker per workspace mount; frecency LMDB under /jackin/state/context/index/. Production-proven: nushell pins =0.9.6. Stable pin unchanged, but 0.9.7-nightly.* prereleases are publishing daily and PR #645 rewrites exactly the per-platform file walker + gitignore matcher — the default-features-off pure-Rust-walker guidance needs a deliberate re-check before any 0.9.7 bump. PORT the bin-only agent-formatting layer (~800 LOC: frecency words, def-first context, import de-noise, cursor pagination) — upstream PR candidate. Caveats: pre-1.0 churn, author-controlled forks (neo_frizbee, notify-debouncer) in the tree, vendored libgit2 + LMDB native deps, notify at RC — cargo-vet the subtree. Note: the crates.io name fff is an unrelated squatter; the crate is fff-search. |
| sem | Rust, MIT/Apache-2.0 | DEPEND (core) + REIMPLEMENT (ledger) | sem-core = "=0.18.0" with per-role lang-* grammar features: entity extraction (38 formats), two-pass entity graph, structural-hash diff (compute_semantic_diff — rename/move/cosmetic detection), budgeted context packer (build_context_result_bounded), orient ranking. API note carried from the 0.16→0.18 jump: 0.16.0 added a public omitted field to ContextResult, breaking struct-literal construction — code against constructors, not literals; 0.17/0.18 add unique-method-name call edges (Python/Ruby), Parent::child qualifiers, and sem orient --pack turn-zero packed briefings (relevant to the read-stub/prefetch layer). Externally consumed already (weave, inspect). The attention ledger / delta-fill machinery lives in bin-first sem-mcp — reimplement natively in the capsule supervisor (~small, spec documented upstream), generalized to all tool traffic. SKIP: sem-cloud (fly.dev egress), telemetry (default-on, opt out regardless), self-update, sem setup git rewiring, the Read/Grep-denying guard hook. Watch: cli/mcp crates.io publishing still wedged — sem-cli/sem-mcp frozen at 0.14.1 while sem-core ships 0.18.0, sem-cloud-client still unpublished; 0.x breaking minors (two in the 19 hours after the 07-03 sweep). |
| mq | Rust, MIT | DEPEND | mq-lang = { version = "=0.6.4", features = ["sync"] } (Send/Sync engine; leave http-import and the new default-off http-import-ureq both off — kills remote-module supply chain), mq-markdown = { version = "=0.6.4", features = ["json"] } (19-crate minimal tree, nushell-proven) for chunking-only paths. Compile canned programs once (Engine::compile/eval_compiled): outline, section, skeleton, code-blocks. Ship the mq binary + its processing-markdown skill in capsule images as the agent-facing escape hatch (0.6.4 adds CLI --count and --skip/--limit result pagination — directly useful for budgeted escape-hatch queries). Caveats: bus factor 1, weekly 0.x releases, ~291-crate tree for full mq-lang (no format gating upstream — a features PR candidate); mq-db is a 0.1.x prototype worth watching for index design, not depending on. |
| RTK | Rust, Apache-2.0 | VENDOR (corpus) + PORT (kernels) | Not a library (no lib.rs; crates.io name taken by an unrelated project; invalid SPDX string blocks upstream publish). Vendor with attribution: the 63 built-in TOML filters (with inline [[tests]] — the single highest-value asset), the 86-rule classifier + shell lexer (incl. quote-aware unattestable-construct guard and transparent_prefixes), the 8-stage TOML pipeline with unless anti-swallow guards, never_worse guard, three-tier ParseResult (+ [N more lines] anti-confusion end-marker contract, per-filter panic isolation, reduced() cap guards, content-sniffing pipe detection, extract_json_object, TOCTOU-safe trust-before-load, integrity sidecars). Port the high-traffic native filters (cargo/git/gh/pytest/go-NDJSON/tsc/vitest) against the StreamFilter trait shape (77 CLI commands / 62 modules upstream — port by traffic, not parity). SKIP: the 6.9k-line installer class (obsoleted by shims), permission mediation, chars/4 ledger, hardcoded vendor savings_pct, telemetry (consent-gated upstream, but its low_savings_commands field can leak command arguments — the engine sends nothing). Bridge option: bake the rtk binary into capsule images and drive rtk pipe --filter / rtk rewrite from shims until native filters land — proven seams, trivially removable. |
| headroom | Python + Rust, Apache-2.0 | REIMPLEMENT to documented spec | No published crates (upstream at v0.30.0 under headroomlabs-ai; workspace crates 0.1.0, path-deps, parity-with-Python constraints baked in; headroom crate name squatted by an audio tool). Reimplement natively using their own documented designs + crate choices: live-zone dispatcher (frozen-floor from cache_control, latest-user-message ceiling, per-type thresholds, token-validation gate, byte-range surgery, per-block manifest), CCR store (exact schema + <<ccr:HASH>> blake3 wire format for interop with the operator's running headroom MCP), typed router (magika ONNX tier optional → unidiff parse-oracle with panic guard → aho-corasick keywords), cache-economics gate (net_mutation_gain, 1.25×/0.1× multipliers), auth-mode policy, holdout counterfactual measurement (conversation-stable blake3 cohorts + online-variance CI), the lossless-compaction tier with runtime round-trip self-check (--lossless mode, shipped upstream in v0.29.0), TextCrusher (deterministic extractive prose scoring — already Rust in headroom-core), the corrected net_mutation_gain (penalty over S+ΔT), and the TOIN observe-only/offline-publish learning architecture with subscription write-quarantine. Their bug classes (#1006 marker-without-tool, #1077 retrieve loop, #1307 lossy tool-output summaries) become the engine's test plan. SKIP: kompress ML models (gated upstream, still default fallback — the engine has no ML tier at all), the Python proxy/memory/graph stack (memory schema: only 2/6 tables agent-scoped, no migrations — counterexample), the max(cli_filtering, rtk) savings collapse (unattributable double-counting), unpinned git-main MCP installs, bundling of RTK/lean-ctx binaries. Evidence note: headroom's "independent 47.5%" was a conflation with its own self-benchmark and its 4.8%-median fleet telemetry is historical (beacon removed) — its evidence tier is now self-report-plus-academic-backing, same class as the others. Keep operator-side headroom-MCP as an A/B comparison arm. |
| lean-ctx | Rust, Apache-2.0 | PORT kernels; optional sidecar binary | Published on crates.io (3.8.18; v3.9.0 cut in-repo 2026-07-04 — AI Gateway team mode, grammar-addon dylibs, agents-only hook default — but unreleased and unpublished) but explicitly no internal-API stability, and the SDK facade is git-only (publish = false) pinning tree-sitter×26 + ort + http into consumers — the monolith in library clothing. Port with attribution: read-stub cache (core/cache.rs + read_stub_index + conversation/task: scoping — highest ROI, smallest dep surface), deterministic crushers (json_crush, tabular/yaml, the 77 shell-pattern modules — count corrected round-2 — as filter references), signatures_ts outline engine (declarative per-language tree-sitter queries; link per-role grammar subsets), CFT Φ-scoring + greedy-knapsack context compiler (pure data model; QUBO spike confirmed greedy suffices), savings-ledger bounce netting (5-tick window, 0.30/≥3 force-full, 10-tick post-edit exclusion). Adopt doctrines: recovery footers on every lossy read, lazy tool surface + doctor overhead --gate (their minimal-arm prefix budget is CI-pinned at 1,835 tokens — ours must beat it), measurement-before-mutation, net-of-injection accounting, corpus-signature freshness, deterministic-argmax learning for cache stability (#498), the mode-default table (docs never compressed; no lossy override), anchored per-line-hash read mode, the Kolmogorov gzip gate, footprint self-ablation (#959). Round-2 count corrections: 13 read modes; release binary measured 75.9 MiB. SKIP: daemon/dashboard/proxy/19 installers/cognition sprawl/in-process ort embeddings/SPLADE (watch)/A2A HMAC transport (upstream's endpoint never verifies)/scent-field stigmergy (novel zero-token multi-agent coordination — record, revisit with the fleet story). Optional: ship the lean-ctx binary as a role addon preconfigured with the faithful-arm profile (tool_profile=minimal, rules_injection=off) for users wanting the full engine — its defaults demonstrably tax harness workloads. |
| codedb | Zig, BSD-3-Clause | DESIGN DONOR (reimplement) | No Rust linkage path worth the FFI cost (Zig 0.16 toolchain, mutex-guarded stateful engine, alpha snapshot format at v0.2.5827, bus factor 1). Take the designs: query-verb grammar + output-shaping idioms (skeleton { … N lines } elision, escalation footers, unchanged:HASH, paths_only, pagination tails, one-line omission markers), value-ordered budget packing (handleContext two-step pack), additive-never-filter ranking stack with per-signal kill switches (eval-tuned on SWE-bench-Lite sweeps), warm-daemon + snapshot economics, agent-registry file-lock primitive. Rust equivalents per capability: tree-sitter (outlines — better than codedb's line heuristics), tantivy or roaring-bitmap trigrams + bm25 (search), fst/nucleo (symbol/fuzzy), ignore+globset (walker), notify (watcher — better than 2s polling), xxhash-rust/blake3 (change detection), memmap2+rkyv (snapshots), petgraph (call graph + PageRank), gix (co-change, no shell-out), rmcp (optional MCP facade). Round-2 additions: config-fingerprinted cache keys, ConvergenceGovernor anti-loop ring, handler-path-exact warmup replay, the reader.md hardening set (traversal rejection, entry caps, LOC ceiling, golden hash-parity tests, length-gated prepend), the failed-experiments ledger practice, two-stage mtime→hash change detection, per-response token cost as a benchmark axis, and the OpenAI-strict oneOf lesson (keep composition server-side). Anti-lessons from its own honest benchmarks: plain FTS5 beat it end-to-end once (efficiency > structural richness); don't hold full contents in RAM; gate every smart feature on a measured eval. Its install surface is the trust-boundary anti-pattern spec: undocumented global command-blocking hook, self-granted permissions, competitor hook-fronting, default-on telemetry uploaded to the vendor on session close — the engine does none of these by construction. |
| caveman | JS plugin (prompts), MIT | NATIVE PROMPT PACK | No code to reuse — the mechanism is instruction text. Adopt the register design (graduated levels, verbatim guards, auto-clarity carve-outs, subagent report registers) as role-generated policy with the corrected numbers (58.5% ultra on visible prose; ~50% over terse control; format-enforcement does the work, politeness asks do nothing); upstream v1.9.1 has itself retired the ~75% headline for a flat "65% average" — the engine's policy keeps the measured numbers regardless. SKIP: hooks, ~940-token skill listing, per-turn re-injection, wenyan, /caveman-stats (superseded by the ledger), and /caveman-commit-style compression of commit/PR text (see caveman-code's exemption — the correct policy). |
| caveman-code | TS coding agent (pi-code fork), MIT | CONSTANTS + HARNESS DONOR | Not a dependency (it's a competing agent). Take: the register exception list (code, commit messages, PR descriptions, security/destructive text — exempt at every level); the per-tool line-budget constants (bash 80 head50/tail30, read 300/200/100, grep 120/80/40, fallback 150) as the L2 defense-in-depth tier; read-dedup-by-fingerprint stubs (convergent with the sem/lean-ctx attention-ledger design the engine already adopts); the subagent output contract ("file:line over prose, no emojis, terse"); and the honest-bench harness template (real child processes, per-task verify.sh, fresh-token accounting, config-fingerprint column, committed raw artifacts) for the engine's cross-tool A/Bs. Its 1.93×-vs-Codex number is real but full-stack at non-default aggressiveness with −1/−2 task quality — cite with caveats only. |
| cavemem | TS MCP memory (unmaintained), MIT | REQUIREMENTS DONOR (do not depend) | Refuted as shipped: 33–36% measured vs 75% marketed, savings refunded by default re-expansion at read time, FTS vocabulary mismatch, global cross-project retrieval, uncapped unmeasured injection, dead privacy config; unmaintained since 2026-05. Take exactly one artifact — the preserved-segment tokenizer (~100 lines, byte-exact technical-token pass-through, ported as the engine's prose-transform safety contract) — and its four defects as L5 requirements (index canonical form; embed canonical; project-scoped retrieval; budgeted+measured injection). Its ~300-token 4-tool MCP surface is a good leanness example. |
| pxpipe | TypeScript, MIT | METHOD DONOR (no code reuse) | Nothing DEPENDable (npm-only pxpipe-proxy, pre-1.0, ~6.5 weeks old, bus factor 1). Take as L0 prior art: the counterfactual savings-accounting design — per-request count_tokens probe on the original uncompressed body, baseline priced under the same observed cache state as the billed request, negative rows never floored — the anti-K1 gold standard in the surveyed field and exactly the shape the engine's ledger must match. Also take its cache-alignment invariants as design rules for any prompt-mutating feature: never add cache_control markers (relocate the caller's existing one to the end of rewritten stable content), freeze rewritten prefixes at first render, quantize any moving old/new boundary, price mode-flips with symmetric cache-burn terms. Record optical/text-to-image context re-encoding as a harness-gated frontier experiment only, never a default: lossy with a silent-confabulation failure shape (byte-exact strings must never live only in pixels), economics and legibility hostage to provider vision-pricing/resample behavior (its own measurements already flip negative across Opus 4.8 / GPT 5.5), and the whole-prompt-rewrite proxy form violates the engine's INV-8/INV-10-class no-proxy constraints. SKIP: the proxy itself, image re-encoding of anything byte-exact. Full teardown: 17 — pxpipe: design teardown. |
Consolidated dependency delta (workspace additions)
# crates/jackin-context/Cargo.toml (new)
fff-search = { version = "=0.9.6", default-features = false }
fff-query-parser = { version = "=0.9.6", default-features = false }
sem-core = { version = "=0.18.0", default-features = false, features = ["git", "parallel"] } # + per-role lang-* features; ContextResult gained a public `omitted` field in 0.16.0 — construct via constructors, not struct literals
mq-lang = { version = "=0.6.4", features = ["sync"] } # http-import AND http-import-ureq stay off
mq-markdown = { version = "=0.6.4", features = ["json"] }
tree-sitter = "0.26" # + per-role grammar crates (MIT — deny.toml clean)
tiktoken-rs = "0.12" # local estimation only; billing-grade counts via count_tokens API
blake3 = "1"
aho-corasick = "1"
petgraph = "0.8" # optional: call-graph feature
gix = "0.73" # co-change signals, no git shell-out
notify = "8" # watcher (stable release, not the RC fff pins internally)
# storage: turso (already in workspace via jackin-usage)Vendored (with NOTICE attribution, Apache-2.0→Apache-2.0): RTK filters/*.toml corpus + classifier rule table; headroom live-zone/CCR constants and thresholds transcribed from spec; lean-ctx pattern-module references. Explicitly not added: tantivy (revisit only if BM25-over-hashmap postings proves insufficient), ort/embedding runtimes (non-goal), any Python.
Upstream contributions worth filing
- fff: publish the agent-formatting layer as a lib crate (or feature of
fff-search); declarerust-version; CHANGELOG. - sem: split
sem-mcp's cache/ledger into a published library crate; unwedge cli/mcp publishing (sem-cloud-client). - mq: cargo features to gate data-format builtins (shrinks the 291-crate tree); publish
mq-lint. - RTK: fix the invalid
license = "Apache 2.0"SPDX string; publishrtk-filters(TOML corpus + engine) as a crate — the community fork ecosystem shows the demand. - headroom: publish
headroom-coreto crates.io once the parity constraints drop.